7 Commits

Author SHA1 Message Date
340bdbe39e Merge branch 'make-default-target': plain make builds the binary
All checks were successful
check / check (push) Successful in 1m12s
2026-07-24 11:21:18 +07:00
a1c3b852c3 Make the binary the default Make target
All checks were successful
check / check (push) Successful in 4s
Plain make now builds sfdupes (previously the default was all =
check + build); make build remains as an alias, so the Dockerfile
and existing habits keep working. The sfdupes target is phony: go
build's own cache decides what to recompile.
2026-07-24 11:21:16 +07:00
9f03eb3e2a Merge branch 'scan-wide-phases': scan-wide phases, concurrent operands, batched updates
All checks were successful
check / check (push) Successful in 1m8s
2026-07-24 10:26:54 +07:00
3ecf73c80a Make phases scan-wide, walk operands concurrently, batch updates
All checks were successful
check / check (push) Successful in 5s
All PATH operands belong to a single scan: every operand seeds the
shared walk worker pool, and each pass (walk, stat, hash, update)
runs exactly once over the whole scan, so pass totals, percentages,
and ETAs are scan-global. The per-operand walk/hash/update cycles and
their stderr operand announcements are gone; duplicate paths from
overlapping operands are deduplicated before stat.

The update pass now commits in batched transactions (10k changes per
batch) instead of one scan-wide transaction: the filesystem is
authoritative and the database is an eventually-consistent reflection
of it, so scan-level atomicity buys nothing, while batches keep the
WAL small and let concurrent reports observe progress.
2026-07-24 10:26:52 +07:00
732fc351d7 Merge branch 'parallel-phases': sequential phases, parallelism within each
All checks were successful
check / check (push) Successful in 1m9s
2026-07-24 08:12:49 +07:00
1e7a519608 Split the stat pass back out of the walk
All checks were successful
check / check (push) Successful in 4s
Phases are strictly sequential again — walk, stat, hash, update per
operand — with parallelism only inside each phase. The walk
enumerates paths with per-directory workers (no lstat of file
entries); the stat pass lstats every collected path with per-file
workers, restoring its exact-total/ETA progress bar and per-file
parallelism inside wide flat directories.
2026-07-24 08:12:47 +07:00
09ff9b5f30 Merge branch 'scan-operand-progress': announce operands on stderr
All checks were successful
check / check (push) Successful in 1m15s
2026-07-24 07:52:55 +07:00
7 changed files with 372 additions and 183 deletions

View File

@@ -6,13 +6,15 @@ BINARY := sfdupes
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
LDFLAGS := -X main.Version=$(VERSION)
.PHONY: all build test lint fmt fmt-check check docker hooks clean
.PHONY: sfdupes build test lint fmt fmt-check check docker hooks clean
all: check build
build:
# Default target: build the binary. Phony so go build (which has its
# own build cache) always decides what to recompile.
sfdupes:
go build -ldflags "$(LDFLAGS)" -o $(BINARY)
build: sfdupes
test:
@go test -timeout 30s -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \

View File

@@ -130,12 +130,13 @@ All three subcommands operate on a single SQLite database file:
database file is a fatal error (exit 1) telling the user to run
`scan` first.
- The database uses WAL journal mode and a busy timeout, so running a
report while a cron `scan` is in progress is safe; the reports see
the last committed state. Each `PATH` operand's changes are
committed in a single transaction when that operand completes, so
a report never observes a half-finished operand, and a scan that
dies partway keeps every operand completed so far (the interrupted
operand's changes are lost, not corrupted).
report while a cron `scan` is in progress is safe. The filesystem
is authoritative; the database is an eventually-consistent
reflection of it. The update pass applies changes in batched
transactions (keeping the WAL small and letting concurrent reports
observe progress), so a report may see a scan's changes partially
applied, and a scan that dies partway leaves a valid database that
the next scan converges toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
@@ -162,11 +163,12 @@ or a regular file; an operand that does not exist is a fatal error
(exit 1). Because database records persist between runs and are keyed
by absolute path, each operand is resolved to an absolute, lexically
cleaned path (symlinks are not resolved) before walking, so results do
not depend on the working directory. Operands are processed in the
order given, each one walked and committed independently; overlapping
operands (one containing another) are harmless — a file reached via
multiple operands produces one database record (later operands see the
records committed by earlier ones and reuse them unchanged).
not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared
walk worker pool, and every pass runs once over the whole scan, so
pass totals and ETAs are scan-wide. Overlapping operands (one
containing another) are harmless — a file reached via multiple
operands is deduplicated by path and produces one database record.
`scan` synchronizes the database with the filesystem state under the
scanned operands:
@@ -188,35 +190,29 @@ scanned operands:
disjoint trees can be scanned on different schedules into the same
database.
`scan` runs **three sequential passes per operand**, committing each
operand before starting the next:
`scan` runs **four sequential passes over the whole scan**.
Parallelism lives strictly inside each pass; the passes themselves
never overlap:
1. **walk** — enumerate the tree with the worker pool: each worker
reads one directory at a time, records size and mtime for every
regular-file entry (`lstat` while the directory is fresh in
cache), and hands discovered subdirectories back to the shared
queue. Sequential directory enumeration is metadata-latency-bound
and takes hours at tens of millions of files; per-directory
parallelism is what makes the walk tractable on large or busy
pools. Total unknown while running: show a live count, not a
percentage.
2. **hash** — for each new or changed file (per the rules above), read
1. **walk** — enumerate the trees under all `PATH` operands
concurrently with the worker pool: every operand seeds the shared
queue, and each worker reads one directory at a time, collecting
regular-file paths and handing discovered subdirectories back to
the queue. Sequential directory enumeration is
metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable
on large or busy pools. Total unknown while running: show a live
count, not a percentage.
2. **stat** — `lstat` every collected path with the worker pool
(per-file parallelism), recording size and mtime.
3. **hash** — for each new or changed file (per the rules above), read
the first `min(1024, size)` bytes and the last `min(1024, size)`
bytes (the two reads overlap when `size < 2048`; for `size == 0`
hash the empty input) and compute the SHA-256 of each. Unchanged
files are not read and do not appear in this pass's total, which
is therefore exact for meaningful progress and ETA.
3. **update** — apply the operand's insertions, updates, and
deletions to the database in a single transaction.
Because every operand runs its own three passes with its own totals,
`scan` announces each operand on stderr before starting it, so a
multi-operand invocation (e.g. a shell glob) is not mistaken for a
nearly-finished run:
```
scan: /srv/media (operand 3 of 14)
```
4. **update** — apply the scan's insertions, updates, and deletions
to the database in batched transactions.
Rules for the walk:
@@ -238,11 +234,12 @@ Rules for the walk:
records (accepted: the database mirrors what the latest scan could
actually verify).
Concurrency: the walk and hash passes use a worker pool (`--workers`,
default `runtime.NumCPU()`); the walk parallelizes across directories,
so raising `--workers` can speed up metadata-bound walks on busy
pools. The main goroutine owns database writes and progress rendering;
progress display must never block the workers.
Concurrency: the walk, stat, and hash passes each use a worker pool
(`--workers`, default `runtime.NumCPU()`); the walk parallelizes
across directories, stat and hash across files, so raising
`--workers` can speed up metadata-bound passes on busy pools. The
main goroutine owns database writes and progress rendering; progress
display must never block the workers.
`scan` writes nothing to stdout. The summary line on stderr reports the
files seen this run broken down by disposition, plus skips:
@@ -357,8 +354,8 @@ all dupe rows) in human units.
Use the progress-bar library for all scan-pass progress; rendering in the
style of `pv` is the model. All progress goes to stderr.
Each scan pass gets its own bar, repeated per operand. Required
elements for the hash and update passes (known totals):
Each scan pass gets its own bar. Required elements for the stat,
hash, and update passes (known totals):
- elapsed time
- estimated time remaining
@@ -396,7 +393,8 @@ Additional requirements:
The `Makefile` is the single source of truth for all operations:
- `make build` — build the `sfdupes` binary (cgo disabled).
- `make` / `make build` — build the `sfdupes` binary (cgo
disabled); building is the default target.
- `make test` — run the test suite (30-second timeout; reruns with
`-v` on failure).
- `make lint` — run `golangci-lint` with the repo config.

18
TODO.md
View File

@@ -19,6 +19,24 @@
# Completed Steps
- make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes`
(previously it ran `check` plus `build`); `make build` remains as
an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
branch `scan-wide-phases`): all operands seed the shared walk pool
and every pass runs once over the whole scan, so totals and ETAs
are scan-global; the per-operand walk/hash/update cycles and their
stderr announcements are gone; the update pass commits in batched
transactions — the filesystem is authoritative and the database an
eventually-consistent reflection, so scan-level atomicity is not
required
- split the stat pass back out of the walk (2026-07-24, branch
`parallel-phases`): phases are strictly sequential again — walk,
stat, hash, update per operand — with parallelism only inside each
phase; the walk enumerates paths with per-directory workers and the
stat pass lstats them with per-file workers, restoring the exact
total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24,
branch `scan-operand-progress`): with per-operand walk/hash/update
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals

35
db.go
View File

@@ -8,6 +8,7 @@ import (
"io/fs"
"os"
"path/filepath"
"slices"
"strconv"
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
@@ -237,12 +238,40 @@ func loadFileRows(db *sql.DB) ([]scanRec, error) {
return recs, nil
}
// updateBatchSize is the number of record changes committed per
// transaction during the update pass. The filesystem is authoritative
// and the database an eventually-consistent reflection of it, so
// scan-level atomicity is not required; smaller transactions keep the
// WAL small and let concurrent reports observe progress.
const updateBatchSize = 10000
// applyChanges writes one scan's database changes — upserts for new and
// changed files, deletes for vanished ones — in a single transaction,
// so a concurrent report never observes a half-finished scan. Progress
// is rendered on prog (one increment per change).
// changed files, deletes for vanished ones — in batched transactions.
// Progress is rendered on prog (one increment per change).
func applyChanges(db *sql.DB, upserts []scanRec, deletes []string,
prog *progress,
) error {
for batch := range slices.Chunk(upserts, updateBatchSize) {
err := applyBatch(db, batch, nil, prog)
if err != nil {
return err
}
}
for batch := range slices.Chunk(deletes, updateBatchSize) {
err := applyBatch(db, nil, batch, prog)
if err != nil {
return err
}
}
return nil
}
// applyBatch commits one batch of upserts and deletes in a single
// transaction.
func applyBatch(db *sql.DB, upserts []scanRec, deletes []string,
prog *progress,
) error {
ctx := context.Background()

View File

@@ -4,6 +4,7 @@ import (
"context"
"database/sql"
"errors"
"fmt"
"path/filepath"
"slices"
"strings"
@@ -178,3 +179,46 @@ func TestApplyChangesRoundTrip(t *testing.T) {
t.Fatalf("rows = %+v, want just %+v", got, upd)
}
}
func TestApplyChangesBatching(t *testing.T) {
t.Parallel()
db := openTestDB(t)
// One more change than the batch size, so the update spans two
// transactions.
n := updateBatchSize + 1
recs := make([]scanRec, 0, n)
for i := range n {
recs = append(recs, scanRec{
size: int64(i), mtime: 1, head: "h", tail: "t",
path: fmt.Sprintf("/batch/%07d", i),
})
}
err := applyChanges(db, recs, nil, newProgress("update", int64(n)))
if err != nil {
t.Fatalf("applyChanges: %v", err)
}
got, err := loadFileRows(db)
if err != nil || len(got) != n {
t.Fatalf("loadFileRows = %d rows, %v; want %d", len(got), err, n)
}
deletes := make([]string, 0, n)
for _, r := range recs {
deletes = append(deletes, r.path)
}
err = applyChanges(db, nil, deletes, newProgress("update", int64(n)))
if err != nil {
t.Fatalf("applyChanges deletes: %v", err)
}
got, err = loadFileRows(db)
if err != nil || len(got) != 0 {
t.Fatalf("loadFileRows = %d rows, %v; want 0", len(got), err)
}
}

226
scan.go
View File

@@ -99,48 +99,27 @@ type scanStats struct {
skipped int
}
// syncScan synchronizes the database with the filesystem under roots.
// Operands are processed in order, each one walked, hashed, and
// committed independently, so an interrupted scan keeps every operand
// completed so far. Records outside the roots are never touched.
// syncScan synchronizes the database with the filesystem under roots:
// four scan-wide passes — walk, stat, hash, update — each parallel
// internally, run strictly in sequence over all operands together.
// Records outside the roots are never touched.
func syncScan(db *sql.DB, roots []string, workers int,
oneFS bool,
) (scanStats, error) {
var st scanStats
for i, root := range roots {
// Each operand runs its own walk/hash/update sequence, so
// announce it: without this, the per-operand pass totals look
// like the whole run's.
fmt.Fprintf(os.Stderr, "scan: %s (operand %d of %d)\n",
root, i+1, len(roots))
err := syncRoot(db, root, workers, oneFS, &st)
if err != nil {
return st, err
}
}
return st, nil
}
// syncRoot walks one PATH operand, hashes its new and changed files,
// and commits the operand's record insertions, updates, and deletions
// in a single transaction.
func syncRoot(db *sql.DB, root string, workers int, oneFS bool,
st *scanStats,
) error {
existing, err := loadScopedRows(db, root)
existing, err := loadScopedRows(db, roots)
if err != nil {
return err
return st, err
}
walked, walkErrs := walkPass(root, oneFS, workers)
toHash, unchanged := partitionChanged(walked, existing)
paths, walkErrs := walkPass(roots, oneFS, workers)
recs, statErrs := statPass(uniquePaths(paths), workers)
toHash, unchanged := partitionChanged(recs, existing)
hashed, hashErrs := hashPass(toHash, workers)
st.unchanged += len(unchanged)
st.skipped += walkErrs + hashErrs
st.unchanged = len(unchanged)
st.skipped = walkErrs + statErrs + hashErrs
for _, r := range hashed {
if _, ok := existing[r.path]; ok {
@@ -151,15 +130,15 @@ func syncRoot(db *sql.DB, root string, workers int, oneFS bool,
}
deletes := collectDeletes(existing, unchanged, hashed)
st.removed += len(deletes)
st.removed = len(deletes)
return applyPass(db, hashed, deletes)
return st, applyPass(db, hashed, deletes)
}
// loadScopedRows loads the database records whose paths lie under
// root, keyed by path. Records outside the scanned operands belong to
// other trees and are left untouched.
func loadScopedRows(db *sql.DB, root string) (map[string]scanRec, error) {
// loadScopedRows loads the database records whose paths lie under any
// of the scan roots, keyed by path. Records outside the roots belong
// to other trees and are left untouched.
func loadScopedRows(db *sql.DB, roots []string) (map[string]scanRec, error) {
all, err := loadFileRows(db)
if err != nil {
return nil, err
@@ -168,7 +147,7 @@ func loadScopedRows(db *sql.DB, root string) (map[string]scanRec, error) {
scoped := make(map[string]scanRec)
for _, r := range all {
if underRoot(r.path, root) {
if underAnyRoot(r.path, roots) {
scoped[r.path] = r
}
}
@@ -176,6 +155,38 @@ func loadScopedRows(db *sql.DB, root string) (map[string]scanRec, error) {
return scoped, nil
}
// underAnyRoot reports whether path is any of the roots or lies under
// one of them.
func underAnyRoot(path string, roots []string) bool {
for _, root := range roots {
if underRoot(path, root) {
return true
}
}
return false
}
// uniquePaths deduplicates the walked paths, preserving order.
// Overlapping operands can reach the same file more than once, but the
// database keys records by path, so each file is processed once.
func uniquePaths(paths []string) []string {
seen := make(map[string]bool, len(paths))
out := make([]string, 0, len(paths))
for _, p := range paths {
if seen[p] {
continue
}
seen[p] = true
out = append(out, p)
}
return out
}
// underRoot reports whether path is root itself or lies under it. Both
// must be absolute and lexically clean.
func underRoot(path, root string) bool {
@@ -270,23 +281,35 @@ type dirJob struct {
}
// walkEvent is one walk result delivered to the main goroutine: a
// regular file's stat record, or a warning when fail is set.
// regular-file path, or a warning when fail is set.
type walkEvent struct {
rec fileRec
path string
warn string
fail bool
}
// walkPass enumerates every regular file under root with a
// per-directory worker pool, recording size and mtime from lstat
// while each directory is fresh in cache. It never follows symlinks,
// never descends into directories named .zfs, and warns and continues
// on any per-path error. With oneFS set it never descends into a
// directory on a different filesystem than root.
func walkPass(root string, oneFS bool, workers int) ([]fileRec, int) {
// walkPass enumerates every regular file under all roots concurrently
// with a per-directory worker pool; every operand seeds the shared
// queue. It never follows symlinks, never descends into directories
// named .zfs, and warns and continues on any per-path error. With
// oneFS set it never descends into a directory on a different
// filesystem than its root operand.
func walkPass(roots []string, oneFS bool, workers int) ([]string, int) {
prog := newProgress("walk", -1)
recs, initial, errs := seedRoot(root, prog)
var (
paths []string
initial []dirJob
errs int
)
for _, root := range roots {
p, jobs, e := seedRoot(root, prog)
paths = append(paths, p...)
initial = append(initial, jobs...)
errs += e
}
jobs, subdirs, events := startWalkWorkers(workers, oneFS)
dispatchDirs(initial, jobs, subdirs)
@@ -300,22 +323,22 @@ func walkPass(root string, oneFS bool, workers int) ([]fileRec, int) {
continue
}
recs = append(recs, ev.rec)
paths = append(paths, ev.path)
prog.increment()
}
prog.finish()
return recs, errs
return paths, errs
}
// seedRoot turns the PATH operand into the walk's starting state: a
// regular-file operand becomes a record directly, a directory operand
// regular-file operand becomes a path directly, a directory operand
// becomes the initial job, and a symlink or other non-regular operand
// yields nothing (symlinks are never followed, including as
// operands).
func seedRoot(root string, prog *progress) ([]fileRec, []dirJob, int) {
func seedRoot(root string, prog *progress) ([]string, []dirJob, int) {
fi, err := os.Lstat(root)
if err != nil {
prog.warnf("walk %s: %v", root, err)
@@ -333,11 +356,9 @@ func seedRoot(root string, prog *progress) ([]fileRec, []dirJob, int) {
return nil, []dirJob{{path: root, rootDev: dev, rootDevOK: ok}}, 0
case fi.Mode().IsRegular():
rec := fileRec{path: root, size: fi.Size(), mtime: fi.ModTime().Unix()}
prog.increment()
return []fileRec{rec}, nil, 0
return []string{root}, nil, 0
default:
return nil, nil, 0
}
@@ -439,37 +460,12 @@ func walkOneDir(job dirJob, oneFS bool, events chan<- walkEvent) []dirJob {
continue
}
events <- fileEvent(p, e)
events <- walkEvent{path: p}
}
return subs
}
// fileEvent lstats one regular-file directory entry into its walk
// event.
func fileEvent(p string, e fs.DirEntry) walkEvent {
fi, err := e.Info()
switch {
case err != nil:
return walkEvent{
warn: fmt.Sprintf("walk %s: %v", p, err),
fail: true,
}
case !fi.Mode().IsRegular():
return walkEvent{
warn: fmt.Sprintf("walk %s: %v", p, errNotRegular),
fail: true,
}
default:
return walkEvent{rec: fileRec{
path: p,
size: fi.Size(),
mtime: fi.ModTime().Unix(),
}}
}
}
// subdirJob applies the descent rules to directory p: never enter
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
// snapshot), and with -x never enter a directory on a different
@@ -514,6 +510,72 @@ func deviceOfInfo(fi fs.FileInfo) (uint64, bool) {
return statDev(st), true
}
// statPass lstats every collected path in a worker pool, recording size
// and mtime. Paths that fail to stat (or are no longer regular files)
// are warned about and dropped.
func statPass(paths []string, workers int) ([]fileRec, int) {
type result struct {
rec fileRec
err error
}
jobs := make(chan string, workQueueDepth)
results := make(chan result, workQueueDepth)
for range workers {
go func() {
for p := range jobs {
fi, err := os.Lstat(p)
switch {
case err != nil:
results <- result{rec: fileRec{path: p}, err: err}
case !fi.Mode().IsRegular():
results <- result{
rec: fileRec{path: p},
err: errNotRegular,
}
default:
results <- result{rec: fileRec{
path: p,
size: fi.Size(),
mtime: fi.ModTime().Unix(),
}}
}
}
}()
}
go func() {
for _, p := range paths {
jobs <- p
}
close(jobs)
}()
prog := newProgress("stat", int64(len(paths)))
var errs int
recs := make([]fileRec, 0, len(paths))
for range paths {
r := <-results
if r.err != nil {
errs++
prog.warnf("stat %s: %v", r.rec.path, r.err)
} else {
recs = append(recs, r.rec)
}
prog.increment()
}
prog.finish()
return recs, errs
}
// hashResult carries one file's head/tail hashes (or the error that
// prevented hashing it) from the hash workers to the main goroutine.
type hashResult struct {

View File

@@ -110,33 +110,6 @@ func TestHashHeadTailErrors(t *testing.T) {
}
}
// walkedPaths returns the sorted paths of walked records.
func walkedPaths(recs []fileRec) []string {
paths := make([]string, 0, len(recs))
for _, r := range recs {
paths = append(paths, r.path)
}
slices.Sort(paths)
return paths
}
// walkedByPath finds the walked record with the given path.
func walkedByPath(t *testing.T, recs []fileRec, path string) fileRec {
t.Helper()
for _, r := range recs {
if r.path == path {
return r
}
}
t.Fatalf("no walked record for %q", path)
return fileRec{}
}
func TestWalkPass(t *testing.T) {
t.Parallel()
@@ -158,18 +131,15 @@ func TestWalkPass(t *testing.T) {
t.Fatal(err)
}
recs, errs := walkPass(dir, false, 4)
paths, errs := walkPass([]string{dir}, false, 4)
if errs != 0 {
t.Fatalf("errs = %d, want 0", errs)
}
if got := walkedPaths(recs); !slices.Equal(got, want) {
t.Fatalf("paths = %q, want %q", got, want)
}
slices.Sort(paths)
// The walk records lstat sizes and mtimes.
if r := walkedByPath(t, recs, want[0]); r.size != 1 || r.mtime <= 0 {
t.Fatalf("rec = %+v, want size 1 and a positive mtime", r)
if !slices.Equal(paths, want) {
t.Fatalf("paths = %q, want %q", paths, want)
}
}
@@ -191,13 +161,42 @@ func TestWalkPassDeepAndWide(t *testing.T) {
slices.Sort(want)
recs, errs := walkPass(dir, false, 8)
paths, errs := walkPass([]string{dir}, false, 8)
if errs != 0 {
t.Fatalf("errs = %d, want 0", errs)
}
if got := walkedPaths(recs); !slices.Equal(got, want) {
t.Fatalf("walked %d paths, want %d", len(got), len(want))
slices.Sort(paths)
if !slices.Equal(paths, want) {
t.Fatalf("walked %d paths, want %d", len(paths), len(want))
}
}
func TestWalkPassMultipleRoots(t *testing.T) {
t.Parallel()
rootA := t.TempDir()
rootB := t.TempDir()
want := []string{
writeFile(t, rootA, "a1", []byte("1")),
writeFile(t, rootA, "sub/a2", []byte("2")),
writeFile(t, rootB, "b1", []byte("3")),
}
slices.Sort(want)
// Operands are enumerated concurrently by the shared pool; order
// is unspecified.
paths, errs := walkPass([]string{rootA, rootB}, false, 4)
if errs != 0 {
t.Fatalf("errs = %d, want 0", errs)
}
slices.Sort(paths)
if !slices.Equal(paths, want) {
t.Fatalf("paths = %q, want %q", paths, want)
}
}
@@ -215,15 +214,15 @@ func TestWalkPassFileAndSymlinkOperands(t *testing.T) {
}
// A regular-file operand is emitted as itself.
recs, errs := walkPass(f, false, 2)
if errs != 0 || len(recs) != 1 || recs[0].path != f || recs[0].size != 4 {
t.Fatalf("file operand: recs = %+v, errs = %d", recs, errs)
paths, errs := walkPass([]string{f}, false, 2)
if errs != 0 || !slices.Equal(paths, []string{f}) {
t.Fatalf("file operand: paths = %q, errs = %d", paths, errs)
}
// A symlink operand is not followed and yields nothing.
recs, errs = walkPass(link, false, 2)
if errs != 0 || len(recs) != 0 {
t.Fatalf("symlink operand: recs = %+v, errs = %d", recs, errs)
paths, errs = walkPass([]string{link}, false, 2)
if errs != 0 || len(paths) != 0 {
t.Fatalf("symlink operand: paths = %q, errs = %d", paths, errs)
}
}
@@ -237,13 +236,41 @@ func TestWalkPassOneFilesystemSameFS(t *testing.T) {
writeFile(t, dir, "sub/deep/b", []byte("b")),
}
recs, errs := walkPass(dir, true, 4)
paths, errs := walkPass([]string{dir}, true, 4)
if errs != 0 {
t.Fatalf("errs = %d, want 0", errs)
}
if got := walkedPaths(recs); !slices.Equal(got, want) {
t.Fatalf("paths = %q, want %q", got, want)
slices.Sort(paths)
if !slices.Equal(paths, want) {
t.Fatalf("paths = %q, want %q", paths, want)
}
}
func TestStatPass(t *testing.T) {
t.Parallel()
dir := t.TempDir()
a := writeFile(t, dir, "a", pattern(1, 10))
b := writeFile(t, dir, "b", pattern(2, 20))
missing := filepath.Join(dir, "vanished")
recs, errs := statPass([]string{a, b, missing}, 2)
if errs != 1 {
t.Fatalf("errs = %d, want 1 for the vanished file", errs)
}
slices.SortFunc(recs, func(x, y fileRec) int {
return strings.Compare(x.path, y.path)
})
if len(recs) != 2 || recs[0].size != 10 || recs[1].size != 20 {
t.Fatalf("recs = %+v, want sizes 10 and 20", recs)
}
if recs[0].mtime <= 0 || recs[1].mtime <= 0 {
t.Fatalf("recs = %+v, want positive mtimes", recs)
}
}
@@ -623,13 +650,11 @@ func TestSyncScanOverlappingRoots(t *testing.T) {
writeFile(t, dir, "sub/f", pattern(1, 10))
// A file reachable via two overlapping operands yields one
// record: the second operand sees the rows the first operand
// committed and reuses them unchanged, which also proves each
// operand commits before the next starts.
// A file reachable via two overlapping operands is deduplicated
// by path in the shared walk and processed once.
st := syncTree(t, db, dir, filepath.Join(dir, "sub"))
if st != (scanStats{added: 1, unchanged: 1}) {
t.Fatalf("stats = %+v, want 1 added 1 unchanged", st)
if st != (scanStats{added: 1}) {
t.Fatalf("stats = %+v, want 1 added", st)
}
if got := recordPaths(dbRecords(t, db)); len(got) != 1 {
@@ -637,6 +662,17 @@ func TestSyncScanOverlappingRoots(t *testing.T) {
}
}
func TestUniquePaths(t *testing.T) {
t.Parallel()
got := uniquePaths([]string{"/a", "/b", "/a", "/c", "/b"})
want := []string{"/a", "/b", "/c"}
if !slices.Equal(got, want) {
t.Fatalf("uniquePaths = %q, want %q", got, want)
}
}
func TestReportsNeverTouchFilesystem(t *testing.T) {
t.Parallel()