6 Commits
Author SHA1 Message Date
clawbot 92c02c3bd1 Walk an operand nested on another filesystem with -x (closes #106)
check / check (push) Waiting to run
With -x, an operand under another operand is kept and walked on its
own when the walk of the deepest kept operand above it cannot reach
it: some directory from it (from its directory, for a regular file) up
to but not including that operand is on a different filesystem than
that operand. Below a kept operand, the rule keeping records under a
directory -x did not enter covers only directories its own walk did
not enter, so records of files gone from it are deleted.

The new tests stub the devices and share their scan steps with the
test for #99. Comments and
README sentences on operands dropped before the walk now also hold
with -x.

Model: opus-5-5
2026-10-14 21:40:02 +02:00
clawbot 80bd7c0999 Keep records on other filesystems under an operand with -x (closes #99)
check / check (push) Waiting to run
With -x the walk does not enter a directory on another filesystem than
its operand, but the update phase still deleted every record below it.
A separate scan of a filesystem mounted under the operand therefore lost
its records on every -x scan and re-hashed its whole tree.

The walk now sends an event naming each directory it declines this way,
and the walk phase treats the records below those directories like
records outside the operands: they are not deleted, and their sizes join
the size census. README "scan mode" and "Rules for the walk" state the
rule.

Model: opus-5-5
2026-10-14 20:22:50 +02:00
clawbot 5dc6bfb6b7 Open files to hash without blocking and reject non-regular ones (closes #100)
check / check (push) Waiting to run
A regular file replaced by a FIFO after the walk statted it made the
hash phase's open wait forever for a writer, so the scan never ended
and kept the lock. The hash and content phases now open each file with
O_NONBLOCK and check with fstat that it is still a regular file;
anything else is a per-path error, warned about and skipped. The new
test replaces a walked file with a FIFO before the hash phase reads it.

Model: opus-5-5
2026-10-14 20:18:32 +02:00
clawbot 1b26b5ce77 Name the scan lock file from the database's resolved path (closes #101)
check / check (push) Waiting to run
The lock file was the configured database path with .lock appended, so
a scan reaching the database through a symlink to it locked a different
file and ran alongside a scan holding the lock through the real path.
Before it takes the lock, scan now creates the database file, empty, if
it is missing, following a symlink to its target as SQLite would, and
names the lock file from that file's path with symlinks resolved. A
scan started before the database exists therefore takes the same lock
file as every later scan through the same path. testDBPath resolves its
temp dir, so lock errors name the expected path where the temp dir is
itself reached through a symlink.

Model: opus-5-5
2026-10-14 20:11:54 +02:00
clawbot a4d76af8fe Redraw a finished bar below a warning for its last file (closes #102)
check / check (push) Waiting to run
On a terminal, warnf clears the bar's line before it prints a warning.
When the warning is for the last item of a phase, the bar has already
drawn its finished frame and the progress library draws nothing more
for a finished bar, so the warning took the bar's place for good. warnf
now draws the finished bar's last frame again below the warning.
Output when stderr is not a terminal does not change.

Model: opus-5-5
2026-10-14 19:55:58 +02:00
clawbot 5900feb515 Store mtime to the nanosecond so a same-second rewrite is re-hashed (closes #12)
check / check (push) Successful in 1m39s
scan recorded mtime in whole seconds, so a file rewritten in place at
the same size within the same second as its recorded mtime was classed
unchanged and kept its old hashes. The files table keeps mtime as whole
Unix seconds and gains mtime_nsec, the nanoseconds within that second.
scan holds the mtime as a time.Time and decides "newer" by comparing
Unix() and then Nanosecond(), so any time a filesystem can record
compares in the right order; After would misorder one too late for a
time.Time to hold without wrapping. The walk, a file given as an
operand, and the content phase's recheck all move over. PRAGMA
user_version stays 1, per the owner's ruling. README states what both
columns hold.

Model: opus-5-5
2026-10-08 02:51:53 +02:00
11 changed files with 1094 additions and 133 deletions
+57 -31
View File
@@ -271,16 +271,21 @@ All three subcommands operate on a single SQLite database file:
error (exit 1) telling the user to run `scan` first. error (exit 1) telling the user to run `scan` first.
- Only one `scan` runs against a database at a time. For its whole run, `scan` - Only one `scan` runs against a database at a time. For its whole run, `scan`
holds an exclusive `flock(2)` lock on a lock file beside the database, named holds an exclusive `flock(2)` lock on a lock file beside the database, named
by appending `.lock` to the database path (`/var/lib/sfdupes/db.sqlite.lock` by appending `.lock` to the database's path with symlinks resolved
by default), taken before it walks the filesystem or opens the database. A (`/var/lib/sfdupes/db.sqlite.lock` by default), taken before it walks the
second `scan` against the same database does not wait: it fails at once with a filesystem or opens the database. Every path to one database, through a
one-line error naming the lock file and exits 1, without walking anything or symlink to the file or to a directory above it, names the same lock file. Just
opening the database, and the running scan carries on. The lock file is before it takes the lock, `scan` creates the database file, empty, if it is
created on first use, open to its owner only, and left in place: a leftover missing (through a symlink at the database path, it creates the symlink's
file blocks nothing, because the lock ends with the process holding it however target), so the lock file is named the same way before the first scan sets up
it ends, a fatal error or an interrupt included, and deleting the file while a the database as after. A second `scan` against the same database does not
scan runs would let a second scan start. `report` and `trees` never take the wait: it fails at once with a one-line error naming the lock file and exits 1,
lock, so they run during a scan. without walking anything or opening the database, and the running scan carries
on. The lock file is created on first use, open to its owner only, and left in
place: a leftover file blocks nothing, because the lock ends with the process
holding it however it ends, a fatal error or an interrupt included, and
deleting the file while a scan runs would let a second scan start. `report`
and `trees` never take the lock, so they run during a scan.
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so - While `scan` runs, the database is in WAL journal mode with a busy timeout, so
running a report while a cron `scan` is in progress is safe. The filesystem is running a report while a cron `scan` is in progress is safe. The filesystem is
authoritative; the database is an eventually-consistent reflection of it. authoritative; the database is an eventually-consistent reflection of it.
@@ -314,7 +319,8 @@ All three subcommands operate on a single SQLite database file:
CREATE TABLE files ( CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- Unix seconds, from lstat mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples
@@ -323,18 +329,22 @@ All three subcommands operate on a single SQLite database file:
``` ```
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
UTF-8. `mtime` is used only for change detection; it is not part of the UTF-8. `mtime` and `mtime_nsec` hold the file's mtime to the nanosecond:
duplicate key. For a file under 10 MiB `head`, `tail`, and `content` all `mtime` the whole Unix seconds, rounded down, and `mtime_nsec` the
hold the whole-file hash (that range is hashed in full, with no end nanoseconds past that second. Split this way they hold any mtime a
windows); for a larger file `head` and `tail` hold the first- and last-64 filesystem can record, one before 1678 or after 2262 included, which a
KiB hashes and `content` the whole-file or sampled hash. All three are empty single 64-bit count of nanoseconds cannot. They are used only for change
strings when the file has never been hashed because its size was unique as detection and are not part of the duplicate key. For a file under 10 MiB
of the last scan that covered it. For a file of 10 MiB or more, `content` `head`, `tail`, and `content` all hold the whole-file hash (that range is
stays empty until the content phase of a scan (see "`scan` mode" below) has hashed in full, with no end windows); for a larger file `head` and `tail`
read the file. A record with an empty `content` is never part of a duplicate hold the first- and last-64 KiB hashes and `content` the whole-file or
group, though it still defines the file for tree reconstruction. The sampled hash. All three are empty strings when the file has never been
`files_signature` index lets SQLite group the records by signature for hashed because its size was unique as of the last scan that covered it. For
`report` without sorting the whole table. a file of 10 MiB or more, `content` stays empty until the content phase of a
scan (see "`scan` mode" below) has read the file. A record with an empty
`content` is never part of a duplicate group, though it still defines the
file for tree reconstruction. The `files_signature` index lets SQLite group
the records by signature for `report` without sorting the whole table.
### Duplicate detection ### Duplicate detection
@@ -388,7 +398,11 @@ results do not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared walk worker scan and are enumerated concurrently: every operand seeds the shared walk worker
pool. Overlapping operands are harmless — an operand that duplicates another or pool. Overlapping operands are harmless — an operand that duplicates another or
lies under another is dropped before walking, so every file is reached exactly lies under another is dropped before walking, so every file is reached exactly
once and produces one database record. once and produces one database record. With `-x`, though, an operand under
another one is kept and walked on its own when any directory from it (from its
directory, for a regular file) up to but not including the operand covering it
is on a different filesystem than that operand, because that operand's walk does
not enter such a directory and so does not reach it.
An operand that is a symlink (never followed, not even as an operand), socket, An operand that is a symlink (never followed, not even as an operand), socket,
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -397,9 +411,8 @@ drops it from the scanned operands before reading the database. Another operand
beneath it is still scanned. The records stored beneath it are not deleted: they beneath it is still scanned. The records stored beneath it are not deleted: they
are treated like any other record outside the scanned operands, including the are treated like any other record outside the scanned operands, including the
content-phase exception below. If it lies under another operand, they are under content-phase exception below. If it lies under another operand, they are under
that operand instead, and are deleted like any other record there that this scan that operand instead, and are treated like any other record there. This is not
did not verify. This is not an error: a scan whose every operand is dropped an error: a scan whose every operand is dropped walks nothing and exits 0.
walks nothing and exits 0.
`scan` synchronizes the database with the filesystem state under the scanned `scan` synchronizes the database with the filesystem state under the scanned
operands: operands:
@@ -421,6 +434,9 @@ operands:
once its size, `head`, and `tail` match another record's. once its size, `head`, and `tail` match another record's.
- A file whose mtime is newer than recorded, or whose size differs, is processed - A file whose mtime is newer than recorded, or whose size differs, is processed
as if new: re-hashed, or recorded without hashes, per the shared-size rule. as if new: re-hashed, or recorded without hashes, per the shared-size rule.
Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the
filesystem gives it a later mtime than recorded, even within the same second.
- A database record whose path lies under one of the scanned operands but was - A database record whose path lies under one of the scanned operands but was
not successfully processed this run is deleted. This removes records for not successfully processed this run is deleted. This removes records for
deleted files. It also removes records for paths that failed to stat or hash deleted files. It also removes records for paths that failed to stat or hash
@@ -434,6 +450,12 @@ operands:
a `content` hash is read for one, wherever it lies, once its size, `head`, and a `content` hash is read for one, wherever it lies, once its size, `head`, and
`tail` match another record's. If that file is gone or has changed since its `tail` match another record's. If that file is gone or has changed since its
record was written, the record is left as it is. record was written, the record is left as it is.
- With `-x`, a record below a directory the walk did not enter because that
directory is on another filesystem than its operand counts as outside the
scanned operands: it is not deleted, and its size joins the size census. A
filesystem mounted under an operand can therefore be scanned on its own
schedule into the same database. Records below an operand at or inside such a
directory are not kept this way, since that operand's own walk verifies them.
`scan` runs **four sequential phases over the whole scan**. Parallelism lives `scan` runs **four sequential phases over the whole scan**. Parallelism lives
inside each phase; batched database writes begin during the hash phase: inside each phase; batched database writes begin during the hash phase:
@@ -500,15 +522,19 @@ Rules for the walk:
- Filesystem boundaries are crossed by default. With `-x` (long form - Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend `--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each into a directory on a different filesystem than its `PATH` operand; each
operand is bounded by its own filesystem. operand is bounded by its own filesystem. The records below such a directory,
unless they lie below an operand at or inside that directory, count as outside
the scanned operands, as described in "`scan` mode" above: they are not
deleted, and their sizes join the size census.
- On any per-path error (permission denied, file vanished between passes, - On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue. unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were Per-file errors never abort the run; the final summary reports how many were
skipped. As specified above, a skipped path that has a database record from an skipped. As specified above, a skipped path that has a database record from an
earlier scan loses that record, unless it failed only in the content phase, or earlier scan loses that record, unless it failed only in the content phase, or
is an operand dropped before the database was read that lies under no other is an operand dropped before the database was read whose record counts as
operand; an unreadable directory subtree likewise loses its records (accepted: outside the scanned operands; an unreadable directory subtree likewise loses
the database mirrors what the latest scan could actually verify). its records (accepted: the database mirrors what the latest scan could
actually verify).
Concurrency: the walk phase (which also stats files), the hash phase, and the Concurrency: the walk phase (which also stats files), the hash phase, and the
content phase each use a worker pool of `--workers` workers (default content phase each use a worker pool of `--workers` workers (default
+30
View File
@@ -29,6 +29,36 @@
# Completed Steps # Completed Steps
- with `-x`, an operand under another operand is walked on its own instead of
being dropped when a directory from it (from its directory, for a regular
file) up to but not including that operand is on a different filesystem than
that operand, whose walk then cannot reach it, and the records of files gone
from below it are deleted (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/106)
- with `-x`, `scan` keeps the records below a directory on another filesystem
under its operand, and counts their sizes in the size census, instead of
deleting them (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
- the hash and content phases open each file without blocking and check that it
is still a regular file, so one replaced by a FIFO or device node after it was
statted is warned about and skipped instead of hanging the scan (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/100)
- the scan lock file is named from the database's path with symlinks resolved,
after `scan` creates the database file if it is missing, so a scan through a
symlink to the database fails while another scan holds the lock (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/101)
- on a terminal, a warning for the last file of a phase no longer erases the
phase's finished bar: the bar is drawn again below the warning (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/102)
- `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus
`mtime_nsec`, and compares it at that resolution, so a same-size rewrite
within the same second is re-hashed (2026-10-07,
https://git.eeqj.de/sneak/sfdupes/issues/12)
- cut the narration from `TODO.md` Completed Steps and from the comments in - cut the narration from `TODO.md` Completed Steps and from the comments in
`script/` and both Dockerfiles; §Workflow now branches from and merges to `script/` and both Dockerfiles; §Workflow now branches from and merges to
`next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49) `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
+54 -22
View File
@@ -11,6 +11,7 @@ import (
"path/filepath" "path/filepath"
"slices" "slices"
"strconv" "strconv"
"time"
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo // The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
@@ -34,18 +35,25 @@ const schemaVersion = 1
// scan. // scan.
const dbDirPerm = 0o755 const dbDirPerm = 0o755
// dbFilePerm is the mode for a database file created by scan, the mode
// SQLite gives a database file it creates.
const dbFilePerm = 0o644
// lockFilePerm is the mode for the scan lock file. Anyone who can open // lockFilePerm is the mode for the scan lock file. Anyone who can open
// the file can hold the lock and keep every scan from running, so it // the file can hold the lock and keep every scan from running, so it
// is open to its owner only. // is open to its owner only.
const lockFilePerm = 0o600 const lockFilePerm = 0o600
// createTableSQL is the schema applied to a fresh database. Paths are // createTableSQL is the schema applied to a fresh database. Paths are
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. // BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime
// holds whole Unix seconds and mtime_nsec the nanoseconds within that
// second.
const createTableSQL = ` const createTableSQL = `
CREATE TABLE files ( CREATE TABLE files (
path BLOB PRIMARY KEY, path BLOB PRIMARY KEY,
size INTEGER NOT NULL, size INTEGER NOT NULL,
mtime INTEGER NOT NULL, mtime INTEGER NOT NULL,
mtime_nsec INTEGER NOT NULL,
head TEXT NOT NULL, head TEXT NOT NULL,
tail TEXT NOT NULL, tail TEXT NOT NULL,
content TEXT NOT NULL content TEXT NOT NULL
@@ -61,10 +69,11 @@ CREATE INDEX files_signature ON files (size, head, tail, content)
// upsertSQL inserts one file record, replacing any existing record for // upsertSQL inserts one file record, replacing any existing record for
// the same path. // the same path.
const upsertSQL = ` const upsertSQL = `
INSERT INTO files (path, size, mtime, head, tail, content) INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content)
VALUES (?, ?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (path) DO UPDATE SET ON CONFLICT (path) DO UPDATE SET
size = excluded.size, mtime = excluded.mtime, size = excluded.size, mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
head = excluded.head, tail = excluded.tail, head = excluded.head, tail = excluded.tail,
content = excluded.content content = excluded.content
` `
@@ -135,19 +144,37 @@ func openDB(path, params string) (*sql.DB, error) {
// lockScanDatabase takes the lock that keeps a second scan off the // lockScanDatabase takes the lock that keeps a second scan off the
// database at path: an exclusive flock(2) on the file beside it named // database at path: an exclusive flock(2) on the file beside it named
// path with ".lock" appended, created along with the database's parent // by the database's path with symlinks resolved and ".lock" appended,
// directory if missing. A lock held by another scan fails at once // so every path to one database names the same lock file. If missing,
// instead of waiting. The lock lasts until the returned file is closed // the database's parent directory and an empty database file are
// or the process ends. The file is never deleted: a scan that deleted // created first, so the lock file is named the same way before the
// it would let the next scan lock a new file while another still holds // database is set up as after. A lock held by another scan fails at
// the old one. // once instead of waiting. The lock lasts until the returned file is
// closed or the process ends. The lock file is never deleted: a scan
// that deleted it would let the next scan lock a new file while
// another still holds the old one.
func lockScanDatabase(path string) (*os.File, error) { func lockScanDatabase(path string) (*os.File, error) {
err := os.MkdirAll(filepath.Dir(path), dbDirPerm) err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil { if err != nil {
return nil, fmt.Errorf("create database directory: %w", err) return nil, fmt.Errorf("create database directory: %w", err)
} }
lockPath := path + ".lock" // Like SQLite, this follows a symlink at path and creates its
// target. SQLite opens an empty file as a new database.
//nolint:gosec // the operator chooses the database path
dbFile, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, dbFilePerm)
if err != nil {
return nil, fmt.Errorf("create database: %w", err)
}
_ = dbFile.Close()
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return nil, fmt.Errorf("resolve database path: %w", err)
}
lockPath := resolved + ".lock"
//nolint:gosec // the operator chooses the database path //nolint:gosec // the operator chooses the database path
f, err := os.OpenFile(lockPath, os.O_RDWR|os.O_CREATE, lockFilePerm) f, err := os.OpenFile(lockPath, os.O_RDWR|os.O_CREATE, lockFilePerm)
@@ -356,8 +383,8 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
// which is the order of the primary key, so SQLite does not sort. // which is the order of the primary key, so SQLite does not sort.
func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error { func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
rows, err := db.QueryContext(ctx, rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, head, tail, content FROM files "+ "SELECT path, size, mtime, mtime_nsec, head, tail, content "+
"ORDER BY path") "FROM files ORDER BY path")
if err != nil { if err != nil {
return fmt.Errorf("read records: %w", err) return fmt.Errorf("read records: %w", err)
} }
@@ -367,16 +394,18 @@ func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
for rows.Next() { for rows.Next() {
var ( var (
path []byte path []byte
sec, nsec int64
r scanRec r scanRec
) )
err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail, err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail,
&r.content) &r.content)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
r.path = string(path) r.path = string(path)
r.mtime = time.Unix(sec, nsec)
fn(r) fn(r)
} }
@@ -466,10 +495,10 @@ func loadDupeRows(ctx context.Context, db *sql.DB,
// values, and skipping the hash columns keeps the scan's in-memory // values, and skipping the hash columns keeps the scan's in-memory
// index small on multi-million-file databases. // index small on multi-million-file databases.
func loadFileMeta(ctx context.Context, db *sql.DB, func loadFileMeta(ctx context.Context, db *sql.DB,
fn func(path string, size, mtime int64, hashed bool), fn func(path string, size int64, mtime time.Time, hashed bool),
) error { ) error {
rows, err := db.QueryContext(ctx, rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, head <> '' FROM files") "SELECT path, size, mtime, mtime_nsec, head <> '' FROM files")
if err != nil { if err != nil {
return fmt.Errorf("read records: %w", err) return fmt.Errorf("read records: %w", err)
} }
@@ -479,16 +508,16 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
for rows.Next() { for rows.Next() {
var ( var (
path []byte path []byte
size, mtime int64 size, sec, nsec int64
hashed int64 hashed int64
) )
err = rows.Scan(&path, &size, &mtime, &hashed) err = rows.Scan(&path, &size, &sec, &nsec, &hashed)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
fn(string(path), size, mtime, hashed != 0) fn(string(path), size, time.Unix(sec, nsec), hashed != 0)
} }
err = rows.Err() err = rows.Err()
@@ -507,7 +536,7 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
// memory; the rows come ordered by size, head, and tail, so each // memory; the rows come ordered by size, head, and tail, so each
// group's rows arrive together. // group's rows arrive together.
const contentCandidatesSQL = ` const contentCandidatesSQL = `
SELECT f.path, f.size, f.mtime, f.head, f.tail, f.content <> '' SELECT f.path, f.size, f.mtime, f.mtime_nsec, f.head, f.tail, f.content <> ''
FROM files AS f FROM files AS f
JOIN ( JOIN (
SELECT size, head, tail SELECT size, head, tail
@@ -534,16 +563,19 @@ func loadContentCandidates(ctx context.Context, db *sql.DB,
for rows.Next() { for rows.Next() {
var ( var (
path []byte path []byte
sec, nsec int64
r scanRec r scanRec
hashed int64 hashed int64
) )
err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail, &hashed) err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail,
&hashed)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
r.path = string(path) r.path = string(path)
r.mtime = time.Unix(sec, nsec)
fn(r, hashed != 0) fn(r, hashed != 0)
} }
@@ -627,8 +659,8 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
defer func() { _ = st.Close() }() defer func() { _ = st.Close() }()
for _, r := range upserts { for _, r := range upserts {
_, err = st.ExecContext(ctx, _, err = st.ExecContext(ctx, []byte(r.path), r.size,
[]byte(r.path), r.size, r.mtime, r.head, r.tail, r.content) r.mtime.Unix(), r.mtime.Nanosecond(), r.head, r.tail, r.content)
if err != nil { if err != nil {
return fmt.Errorf("upsert %s: %w", r.path, err) return fmt.Errorf("upsert %s: %w", r.path, err)
} }
+20 -7
View File
@@ -10,13 +10,22 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time"
) )
// testDBPath returns a database path inside a fresh temp dir. // testDBPath returns a database path inside a fresh temp dir, with
// symlinks resolved as the scan lock resolves them, so the lock file
// is this path plus ".lock" even where the temp dir is reached through
// a symlink.
func testDBPath(t *testing.T) string { func testDBPath(t *testing.T) string {
t.Helper() t.Helper()
return filepath.Join(t.TempDir(), "db.sqlite") dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
return filepath.Join(dir, "db.sqlite")
} }
// openTestDB creates a fresh scan database in a temp dir. // openTestDB creates a fresh scan database in a temp dir.
@@ -260,10 +269,13 @@ func TestApplyChangesRoundTrip(t *testing.T) {
// written. // written.
recs := []scanRec{ recs := []scanRec{
{ {
size: 2, mtime: 20, head: "h2", tail: "t2", content: "c2", size: 2, mtime: time.Unix(20, 999_999_999), head: "h2", tail: "t2",
path: "/a/tab\tnew\nline", content: "c2", path: "/a/tab\tnew\nline",
},
{
size: 1, mtime: time.Unix(10, 0), head: "h1", tail: "t1",
content: "c1", path: "/a/x",
}, },
{size: 1, mtime: 10, head: "h1", tail: "t1", content: "c1", path: "/a/x"},
} }
err := applyChanges(t.Context(), db, recs, nil, err := applyChanges(t.Context(), db, recs, nil,
@@ -281,7 +293,8 @@ func TestApplyChangesRoundTrip(t *testing.T) {
// An upsert for an existing path updates in place; a delete // An upsert for an existing path updates in place; a delete
// removes exactly its path. // removes exactly its path.
upd := scanRec{ upd := scanRec{
size: 3, mtime: 30, head: "h3", tail: "t3", content: "c3", path: "/a/x", size: 3, mtime: time.Unix(30, 0), head: "h3", tail: "t3", content: "c3",
path: "/a/x",
} }
err = applyChanges(t.Context(), db, []scanRec{upd}, err = applyChanges(t.Context(), db, []scanRec{upd},
@@ -308,7 +321,7 @@ func TestApplyChangesBatching(t *testing.T) {
recs := make([]scanRec, 0, n) recs := make([]scanRec, 0, n)
for i := range n { for i := range n {
recs = append(recs, scanRec{ recs = append(recs, scanRec{
size: int64(i), mtime: 1, head: "h", tail: "t", size: int64(i), mtime: time.Unix(1, 0), head: "h", tail: "t",
path: fmt.Sprintf("/batch/%07d", i), path: fmt.Sprintf("/batch/%07d", i),
}) })
} }
+93 -4
View File
@@ -822,7 +822,7 @@ func holdScanLock(t *testing.T, path string) {
func TestRunSecondScanFails(t *testing.T) { func TestRunSecondScanFails(t *testing.T) {
// README §Database: while one scan holds the lock, a second scan // README §Database: while one scan holds the lock, a second scan
// fails at once, naming the lock file, without creating the // fails at once, naming the lock file, without opening the
// database. // database.
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
@@ -848,10 +848,99 @@ func TestRunSecondScanFails(t *testing.T) {
t.Errorf("stdout = %q, want nothing (data only)", got) t.Errorf("stdout = %q, want nothing (data only)", got)
} }
_, err := os.Stat(path) // Taking the lock created the database file, empty; opening it as
if !errors.Is(err, fs.ErrNotExist) { // a database would have written to it.
t.Errorf("stat %s = %v, want the database not created", path, err) info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
} }
if info.Size() != 0 {
t.Errorf("database size = %d, want 0 (not opened)", info.Size())
}
}
// assertSecondScanFails runs a scan against the database
// SFDUPES_DATABASE names while the test holds the lock on it. It fails
// the test unless that scan fails at once, naming the lock file beside
// path, the database's real path.
func assertSecondScanFails(t *testing.T, path string) {
t.Helper()
var stderr bytes.Buffer
code := run([]string{cmdScan, t.TempDir()}, io.Discard, &stderr)
if code != exitFatal {
t.Errorf("run(scan) = %d, want %d", code, exitFatal)
}
want := "sfdupes: another scan is running (lock held on " +
path + ".lock)\n"
if got := stderr.String(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
}
func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
// README §Database: every path to one database names the same lock
// file, so while a scan holds it through one path, a scan through
// another fails, naming the lock file beside the database's real
// path.
t.Run("symlink to the database", func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
scanFixture(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
holdScanLock(t, path)
t.Setenv(databaseEnv, link)
assertSecondScanFails(t, path)
})
t.Run("symlink to its directory, no database yet", func(t *testing.T) {
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link")
err := os.Symlink(filepath.Dir(path), link)
if err != nil {
t.Fatal(err)
}
holdScanLock(t, path)
t.Setenv(databaseEnv, filepath.Join(link, filepath.Base(path)))
assertSecondScanFails(t, path)
})
t.Run("symlink whose target does not exist yet", func(t *testing.T) {
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
// The first scan takes the lock through the link before the
// database exists, then creates the database through it.
holdScanLock(t, link)
db, err := openScanDatabase(t.Context(), link)
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, link)
t.Setenv(databaseEnv, link)
assertSecondScanFails(t, path)
})
} }
func TestRunScanReleasesLock(t *testing.T) { func TestRunScanReleasesLock(t *testing.T) {
+7
View File
@@ -139,6 +139,13 @@ func (p *progress) warnf(format string, args ...any) {
} }
fmt.Fprintln(os.Stderr, msg) fmt.Fprintln(os.Stderr, msg)
// The library never draws a finished bar again, so the warning for
// a pass's last item would erase it for good: draw its last frame
// again below the warning.
if p.bar != nil && p.bar.IsFinished() {
fmt.Fprint(os.Stderr, p.bar.String())
}
} }
// finish terminates the pass's display. A bar whose pass stopped short // finish terminates the pass's display. A bar whose pass stopped short
+22
View File
@@ -187,3 +187,25 @@ func TestBarStoppedShortKeepsCount(t *testing.T) {
t.Errorf("terminal shows %q, want a count of 2 of 10", shown) t.Errorf("terminal shows %q, want a count of 2 of 10", shown)
} }
} }
// TestWarningAfterLastItemKeepsBar checks that a warning printed after
// a pass's last item, as for a failed read of the last file, leaves the
// pass's finished bar on the terminal below the warning.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestWarningAfterLastItemKeepsBar(t *testing.T) {
stderr := captureStderr(t)
p := &progress{
label: "hash", total: 2, start: time.Now(),
bar: newBar("hash", 2),
}
p.increment()
p.increment()
p.warnf("warning")
p.finish()
if shown := lastFrame(stderr()); !strings.Contains(shown, "(2/2,") {
t.Errorf("terminal shows %q, want a count of 2 of 2", shown)
}
}
+2 -1
View File
@@ -7,6 +7,7 @@ import (
"io" "io"
"os" "os"
"strings" "strings"
"time"
) )
// ioBufSize is the buffer size for the buffered stdout writers. // ioBufSize is the buffer size for the buffered stdout writers.
@@ -21,7 +22,7 @@ const minGroupSize = 2
// only and used by scan for change detection. // only and used by scan for change detection.
type scanRec struct { type scanRec struct {
size int64 size int64
mtime int64 mtime time.Time
head string head string
tail string tail string
content string content string
+9 -2
View File
@@ -11,6 +11,7 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time"
) )
// awkwardDir is a directory name holding every byte the reports escape. // awkwardDir is a directory name holding every byte the reports escape.
@@ -313,8 +314,14 @@ func TestDupeGroupsMtimeExcluded(t *testing.T) {
// mtime is informational only; records differing only in mtime // mtime is informational only; records differing only in mtime
// still group together. // still group together.
recs := []scanRec{ recs := []scanRec{
{size: 9, mtime: 100, head: "h", tail: "t", content: "c", path: "/m/1"}, {
{size: 9, mtime: 200, head: "h", tail: "t", content: "c", path: "/m/2"}, size: 9, mtime: time.Unix(100, 0), head: "h", tail: "t",
content: "c", path: "/m/1",
},
{
size: 9, mtime: time.Unix(200, 0), head: "h", tail: "t",
content: "c", path: "/m/2",
},
} }
groups := dupeGroupsOf(t, recs) groups := dupeGroupsOf(t, recs)
+174 -34
View File
@@ -17,6 +17,7 @@ import (
"strings" "strings"
"sync" "sync"
"syscall" "syscall"
"time"
) )
// The duplicate ladder (see hashSignature and README "Duplicate // The duplicate ladder (see hashSignature and README "Duplicate
@@ -68,7 +69,7 @@ var errInterrupted = errors.New("scan interrupted")
type fileRec struct { type fileRec struct {
path string path string
size int64 size int64
mtime int64 mtime time.Time
dev uint64 dev uint64
ino uint64 ino uint64
} }
@@ -79,7 +80,7 @@ type fileRec struct {
// they would dominate the scan's memory. // they would dominate the scan's memory.
type fileMeta struct { type fileMeta struct {
size int64 size int64
mtime int64 mtime time.Time
hashed bool hashed bool
} }
@@ -202,10 +203,16 @@ func resolveRoots(roots []string) ([]string, error) {
} }
// pruneRoots drops operands already covered by another operand: // pruneRoots drops operands already covered by another operand:
// duplicates and any operand lying under another one. Every remaining // duplicates and any operand lying under another one. With -x, an
// file is then reachable through exactly one root, so walked paths are // operand under another one is kept when the walk of the deepest kept
// unique without keeping a scan-wide set of every path seen. // operand above it cannot reach it: when device puts any directory from
func pruneRoots(roots []string) []string { // it (from its directory, for a regular file) up to but not including
// that operand on a different filesystem than that operand. Every
// remaining file is then reachable through exactly one root, so walked
// paths are unique without keeping a scan-wide set of every path seen.
func pruneRoots(roots []string, oneFS bool,
device func(path string) (uint64, bool),
) []string {
// Shorter-first ordering guarantees an ancestor is kept before any // Shorter-first ordering guarantees an ancestor is kept before any
// operand under it is considered. // operand under it is considered.
sorted := slices.Clone(roots) sorted := slices.Clone(roots)
@@ -220,14 +227,58 @@ func pruneRoots(roots []string) []string {
kept := make([]string, 0, len(sorted)) kept := make([]string, 0, len(sorted))
for _, r := range sorted { for _, r := range sorted {
if !underAnyRoot(r, kept) { above := deepestRoot(r, kept)
if above == "" {
kept = append(kept, r) kept = append(kept, r)
continue
}
// Without -x, and always for a duplicate, r is dropped.
if !oneFS || above == r {
continue
}
aboveDev, aboveOK := device(above)
if !aboveOK {
continue
}
// The walk of above enters a directory only when it is on
// above's filesystem, and then reaches every regular file in it
// whatever device the file reports, so for a regular-file
// operand the directories checked start at its directory.
dir := r
fi, err := os.Lstat(r)
if err == nil && fi.Mode().IsRegular() {
dir = filepath.Dir(r)
}
for ; dir != above; dir = filepath.Dir(dir) {
dev, ok := device(dir)
if ok && dev != aboveDev {
kept = append(kept, r)
break
}
} }
} }
return kept return kept
} }
// deviceOfPath returns the filesystem device ID of the file at path,
// when lstat succeeds and the platform exposes one.
func deviceOfPath(path string) (uint64, bool) {
fi, err := os.Lstat(path)
if err != nil {
return 0, false
}
return deviceOfInfo(fi)
}
// scanStats summarizes one scan's database synchronization for the // scanStats summarizes one scan's database synchronization for the
// final stderr summary, or for the line an interrupted scan prints. // final stderr summary, or for the line an interrupted scan prints.
type scanStats struct { type scanStats struct {
@@ -287,21 +338,22 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
// size, head, and tail match another record's). Records outside the // size, head, and tail match another record's). Records outside the
// roots are never touched, except that the content phase fills in // roots are never touched, except that the content phase fills in
// their content hash. Operands the walk cannot start from are dropped // their content hash. Operands the walk cannot start from are dropped
// first, so the records beneath them count as outside the roots unless // first, so the records beneath them are treated as if those operands
// they lie under another root. // had not been given.
func (s *scanState) runPhases(ctx context.Context, roots []string, func (s *scanState) runPhases(ctx context.Context, roots []string,
workers int, oneFS bool, workers int, oneFS bool,
) error { ) error {
// Types are checked before pruning so that an operand under a // Types are checked before pruning so that an operand under a
// dropped one is still scanned, not dropped as lying under it. // dropped one is still scanned, not dropped as lying under it.
roots = pruneRoots(s.walkableRoots(roots)) roots = pruneRoots(s.walkableRoots(roots), oneFS, deviceOfPath)
err := s.loadIndex(ctx, roots) err := s.loadIndex(ctx, roots)
if err != nil { if err != nil {
return err return err
} }
changed, unhashed := s.walkPhase(startWalk(ctx, roots, oneFS, workers)) changed, unhashed := s.walkPhase(
startWalk(ctx, roots, oneFS, workers), roots)
// A cancelled walk stops early, so its size census covers only part // A cancelled walk stops early, so its size census covers only part
// of the roots, and every file it never reached would look vanished // of the roots, and every file it never reached would look vanished
@@ -329,10 +381,10 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// walkableRoots returns the operands the walk can start from: regular // walkableRoots returns the operands the walk can start from: regular
// files, and directories not named .zfs. Every other operand is warned // files, and directories not named .zfs. Every other operand is warned
// about, counted as skipped, and dropped. A dropped operand is no // about, counted as skipped, and dropped. A dropped operand is no
// longer a root, so the records stored beneath it count as outside the // longer a root, so the records stored beneath it are treated as if it
// roots and are not deleted as unverified, unless it lies under another // had not been given: those under no other root count as outside the
// root. An operand that fails lstat here is kept, and the walk warns // roots and are not deleted as unverified. An operand that fails lstat
// about it. // here is kept, and the walk warns about it.
func (s *scanState) walkableRoots(roots []string) []string { func (s *scanState) walkableRoots(roots []string) []string {
kept := make([]string, 0, len(roots)) kept := make([]string, 0, len(roots))
@@ -369,7 +421,7 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
s.existing = make(map[string]fileMeta) s.existing = make(map[string]fileMeta)
return loadFileMeta(ctx, s.db, return loadFileMeta(ctx, s.db,
func(path string, size, mtime int64, hashed bool) { func(path string, size int64, mtime time.Time, hashed bool) {
prog.increment() prog.increment()
if underAnyRoot(path, roots) { if underAnyRoot(path, roots) {
@@ -390,15 +442,27 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// (the content phase may still fill in its content hash). It returns // (the content phase may still fill in its content hash). It returns
// the new-or-changed files and the unchanged files whose records lack // the new-or-changed files and the unchanged files whose records lack
// hashes; both remain candidates until the census decides whether // hashes; both remain candidates until the census decides whether
// their sizes are shared. // their sizes are shared. A record below a directory -x did not enter
// counts as outside the roots: it is not deleted, and its size joins
// the census. That does not hold below a root at or inside that
// directory, since that root's own walk verifies the records below it.
func (s *scanState) walkPhase( func (s *scanState) walkPhase(
events <-chan walkEvent, events <-chan walkEvent, roots []string,
) ([]fileRec, []fileRec) { ) ([]fileRec, []fileRec) {
prog := newProgress("walk", -1) prog := newProgress("walk", -1)
var changed, unhashed []fileRec var (
changed, unhashed []fileRec
otherFSDirs []string
)
for ev := range events { for ev := range events {
if ev.otherFSDir != "" {
otherFSDirs = append(otherFSDirs, ev.otherFSDir)
continue
}
if ev.fail { if ev.fail {
s.st.skipped++ s.st.skipped++
@@ -415,7 +479,7 @@ func (s *scanState) walkPhase(
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
switch { switch {
case !ok || old.size != ev.rec.size || old.mtime < ev.rec.mtime: case !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime):
changed = append(changed, ev.rec) changed = append(changed, ev.rec)
case old.hashed: case old.hashed:
delete(s.existing, ev.rec.path) delete(s.existing, ev.rec.path)
@@ -428,6 +492,19 @@ func (s *scanState) walkPhase(
prog.finish() prog.finish()
for path, old := range s.existing {
// Kept only when a directory -x did not enter lies below the
// deepest root above the record. A root at or below that
// directory was walked on its own and did not find the file.
// Both contain the record, so the longer path is the deeper.
otherFSDir := deepestRoot(path, otherFSDirs)
if len(otherFSDir) > len(deepestRoot(path, roots)) {
delete(s.existing, path)
s.sizes = append(s.sizes, old.size)
}
}
return changed, unhashed return changed, unhashed
} }
@@ -807,7 +884,7 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
} }
if !fi.Mode().IsRegular() || fi.Size() != r.size || if !fi.Mode().IsRegular() || fi.Size() != r.size ||
fi.ModTime().Unix() > r.mtime { mtimeAfter(fi.ModTime(), r.mtime) {
return fileRec{}, false, nil return fileRec{}, false, nil
} }
@@ -818,6 +895,16 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
}, true, nil }, true, nil
} }
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under // underAnyRoot reports whether path is any of the roots or lies under
// one of them. // one of them.
func underAnyRoot(path string, roots []string) bool { func underAnyRoot(path string, roots []string) bool {
@@ -845,6 +932,20 @@ func underRoot(path, root string) bool {
return strings.HasPrefix(path, prefix) return strings.HasPrefix(path, prefix)
} }
// deepestRoot returns the longest of the roots that path is or lies
// under, or "" when it lies under none.
func deepestRoot(path string, roots []string) string {
deepest := ""
for _, root := range roots {
if underRoot(path, root) && len(root) > len(deepest) {
deepest = root
}
}
return deepest
}
// dirJob is one directory awaiting traversal by the walk workers. It // dirJob is one directory awaiting traversal by the walk workers. It
// carries its operand's filesystem device so -x can stop at // carries its operand's filesystem device so -x can stop at
// filesystem boundaries. // filesystem boundaries.
@@ -855,19 +956,23 @@ type dirJob struct {
} }
// walkEvent is one walk result delivered to the walk phase: a regular // walkEvent is one walk result delivered to the walk phase: a regular
// file's statted record, or a warning when fail is set. // file's statted record, a warning when fail is set, or, when
// otherFSDir is set, a directory -x did not enter because it is on
// another filesystem than its operand.
type walkEvent struct { type walkEvent struct {
rec fileRec rec fileRec
warn string warn string
fail bool fail bool
otherFSDir string
} }
// startWalk seeds every root into the shared walk worker pool and // startWalk seeds every root into the shared walk worker pool and
// returns the event stream: one record per regular file, one warning // returns the event stream: one record per regular file, one warning
// event per per-path error. The channel is closed when the walk // event per per-path error, and one event per directory -x did not
// completes, and also when ctx is cancelled — every goroutine in the // enter. The channel is closed when the walk completes, and also when
// pool abandons its blocking send in that case, so the consumer sees a // ctx is cancelled — every goroutine in the pool abandons its blocking
// truncated but properly terminated stream instead of a stalled one. // send in that case, so the consumer sees a truncated but properly
// terminated stream instead of a stalled one.
func startWalk(ctx context.Context, roots []string, oneFS bool, func startWalk(ctx context.Context, roots []string, oneFS bool,
workers int, workers int,
) <-chan walkEvent { ) <-chan walkEvent {
@@ -934,7 +1039,8 @@ func operandWarning(root string, fi fs.FileInfo) string {
// operand becomes an initial job. walkableRoots has already dropped // operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since // every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records // is warned about and skipped here; it is still a root, so the records
// stored beneath it are deleted as unverified. // stored beneath it, other than those under another root, are deleted
// as unverified.
func seedRoot(ctx context.Context, root string, func seedRoot(ctx context.Context, root string,
events chan<- walkEvent, events chan<- walkEvent,
) []dirJob { ) []dirJob {
@@ -966,7 +1072,7 @@ func seedRoot(ctx context.Context, root string,
sendEvent(ctx, events, walkEvent{rec: fileRec{ sendEvent(ctx, events, walkEvent{rec: fileRec{
path: root, path: root,
size: fi.Size(), size: fi.Size(),
mtime: fi.ModTime().Unix(), mtime: fi.ModTime(),
dev: dev, dev: dev,
ino: ino, ino: ino,
}}) }})
@@ -1124,7 +1230,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
sendEvent(ctx, events, walkEvent{rec: fileRec{ sendEvent(ctx, events, walkEvent{rec: fileRec{
path: p, path: p,
size: info.Size(), size: info.Size(),
mtime: info.ModTime().Unix(), mtime: info.ModTime(),
dev: dev, dev: dev,
ino: ino, ino: ino,
}}) }})
@@ -1133,7 +1239,9 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
// subdirJob applies the descent rules to directory p: never enter // subdirJob applies the descent rules to directory p: never enter
// .zfs (ZFS snapshot pseudo-dirs would list every file once per // .zfs (ZFS snapshot pseudo-dirs would list every file once per
// snapshot), and with -x never enter a directory on a different // snapshot), and with -x never enter a directory on a different
// filesystem than its operand. // filesystem than its operand, sending an event that names it so the
// scan keeps the records below it, except those below an operand at or
// inside it.
func subdirJob(ctx context.Context, p string, e fs.DirEntry, func subdirJob(ctx context.Context, p string, e fs.DirEntry,
parent dirJob, oneFS bool, events chan<- walkEvent, parent dirJob, oneFS bool, events chan<- walkEvent,
) (dirJob, bool) { ) (dirJob, bool) {
@@ -1157,6 +1265,8 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
} }
if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev { if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
sendEvent(ctx, events, walkEvent{otherFSDir: p})
return dirJob{}, false return dirJob{}, false
} }
@@ -1298,6 +1408,38 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
} }
} }
// errNotRegular reports a path that is no longer a regular file when it
// is opened to be read.
var errNotRegular = errors.New("not a regular file")
// openRegularFile opens path for reading and returns it only if it is
// still a regular file, since the path may have been replaced after it
// was statted. O_NONBLOCK keeps the open of a FIFO or a device node
// from waiting forever, and fstat then rejects it. Reads of a regular
// file ignore O_NONBLOCK.
func openRegularFile(path string) (*os.File, error) {
//nolint:gosec // hashing operator-supplied paths is the tool's purpose
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return nil, err
}
fi, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, err
}
if !fi.Mode().IsRegular() {
_ = f.Close()
return nil, errNotRegular
}
return f, nil
}
// emptyHash is the lowercase-hex SHA-256 of the empty input: the head, // emptyHash is the lowercase-hex SHA-256 of the empty input: the head,
// tail, and content hash of every zero-length file. // tail, and content hash of every zero-length file.
const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" + const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
@@ -1320,8 +1462,7 @@ func hashSignature(path string, size int64) (string, string, string, error) {
return emptyHash, emptyHash, emptyHash, nil return emptyHash, emptyHash, emptyHash, nil
} }
//nolint:gosec // hashing operator-supplied paths is the tool's purpose f, err := openRegularFile(path)
f, err := os.Open(path)
if err != nil { if err != nil {
return "", "", "", err return "", "", "", err
} }
@@ -1352,8 +1493,7 @@ func hashSignature(path string, size int64) (string, string, string, error) {
// tail are returned empty, because the content phase keeps the ones its // tail are returned empty, because the content phase keeps the ones its
// records already hold. // records already hold.
func hashContentOnly(path string, size int64) (string, string, string, error) { func hashContentOnly(path string, size int64) (string, string, string, error) {
//nolint:gosec // hashing operator-supplied paths is the tool's purpose f, err := openRegularFile(path)
f, err := os.Open(path)
if err != nil { if err != nil {
return "", "", "", err return "", "", "", err
} }
+605 -11
View File
@@ -20,6 +20,8 @@ import (
"syscall" "syscall"
"testing" "testing"
"time" "time"
"golang.org/x/sys/unix"
) )
// writeFile creates a file with the given content and returns its path. // writeFile creates a file with the given content and returns its path.
@@ -581,6 +583,65 @@ func TestScanContentHashedStalePartners(t *testing.T) {
} }
} }
// TestScanContentSameSecondRewrite is TestScanContentStalePartners for
// a stored file rewritten in place at the same size with an mtime later
// in the same second than recorded: the file counts as changed, so
// neither it nor its match inside the operand is read.
func TestScanContentSameSecondRewrite(t *testing.T) {
t.Parallel()
db := openTestDB(t)
dirA := t.TempDir()
changed := sparseFileWithoutMatch(t, dirA, "changed", headTailMin)
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(changed, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dirA)
before := dbRecords(t, db)
// Rewrite one byte in place, keeping the size.
pokeAt(t, changed, headTailMin/2, []byte{1})
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(changed, later, later)
if err != nil {
t.Fatal(err)
}
dirB := t.TempDir()
sparseFile(t, dirB, "changed-copy", headTailMin)
st := syncTree(t, db, dirB)
if st != (scanStats{walked: 1, added: 1}) {
t.Errorf("stats = %+v, want 1 added and nothing skipped", st)
}
recs := dbRecords(t, db)
for _, r := range recs {
if r.content != "" {
t.Errorf("%s: content = %q, want none: its only match is stale",
r.path, r.content)
}
}
for _, old := range before {
if r := recordByPath(t, recs, old.path); r != old {
t.Errorf("record = %+v, want it left as %+v", r, old)
}
}
if groups := dupeGroups(t, db); len(groups) != 0 {
t.Errorf("groups = %+v, want none", groups)
}
}
// TestScanContentReadFailure checks that a failed content read is // TestScanContentReadFailure checks that a failed content read is
// counted as skipped and leaves the record without a content hash, and // counted as skipped and leaves the record without a content hash, and
// that a later scan tries the read again. // that a later scan tries the read again.
@@ -783,8 +844,8 @@ func TestWalk(t *testing.T) {
t.Errorf("%s: size = %d, want 1..3", r.path, r.size) t.Errorf("%s: size = %d, want 1..3", r.path, r.size)
} }
if r.mtime <= 0 { if r.mtime.Unix() <= 0 {
t.Errorf("%s: mtime = %d, want positive", r.path, r.mtime) t.Errorf("%s: mtime = %v, want after 1970", r.path, r.mtime)
} }
} }
} }
@@ -936,7 +997,7 @@ func dirEntryFor(t *testing.T, dir, name string) fs.DirEntry {
} }
// callSubdirJob runs subdirJob against e under parent, collecting any // callSubdirJob runs subdirJob against e under parent, collecting any
// warning events it emits (subdirJob emits at most one). // events it emits (subdirJob emits at most one).
func callSubdirJob(t *testing.T, p string, e fs.DirEntry, func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
parent dirJob, oneFS bool, parent dirJob, oneFS bool,
) (dirJob, bool, []walkEvent) { ) (dirJob, bool, []walkEvent) {
@@ -1014,8 +1075,15 @@ func TestSubdirJobOneFilesystem(t *testing.T) {
t.Fatalf("accepted = %v, want %v", ok, tc.wantOK) t.Fatalf("accepted = %v, want %v", ok, tc.wantOK)
} }
if len(evs) != 0 { // A refused descent names the directory, so the scan keeps
t.Fatalf("unexpected events: %+v", evs) // the records below it.
var wantEvs []walkEvent
if !ok {
wantEvs = []walkEvent{{otherFSDir: sub}}
}
if !slices.Equal(evs, wantEvs) {
t.Fatalf("events = %+v, want %+v", evs, wantEvs)
} }
// The accepted job must carry the operand's device down, or -x // The accepted job must carry the operand's device down, or -x
@@ -1067,6 +1135,296 @@ func TestSubdirJobStatError(t *testing.T) {
} }
} }
// syncOneFilesystem scans roots into db with -x and returns the scan
// stats. Every device pruneRoots looks up, and the device each
// operand's walk compares its subdirectories with, comes from devices
// instead of from lstat, so a test can put a path on a filesystem that
// no directory here is on. Unlike runPhases, it skips walkableRoots,
// the check for a cancelled walk, and the content phase.
func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
devices map[string]uint64,
) scanStats {
t.Helper()
device := func(path string) (uint64, bool) {
dev, ok := devices[path]
return dev, ok
}
roots = pruneRoots(roots, true, device)
ctx := t.Context()
s := &scanState{db: db}
err := s.loadIndex(ctx, roots)
if err != nil {
t.Fatal(err)
}
jobs, subdirs, events := startWalkWorkers(ctx, 2, true)
// seedRoot emits a regular-file operand itself and returns a job
// only for a directory operand.
var initial []dirJob
for _, root := range roots {
for _, job := range seedRoot(ctx, root, events) {
job.rootDev = devices[root]
initial = append(initial, job)
}
}
dispatchDirs(ctx, initial, jobs, subdirs)
changed, unhashed := s.walkPhase(events, roots)
s.partition(changed, unhashed)
err = s.hashPhase(ctx, 2)
if err != nil {
t.Fatal(err)
}
err = s.updatePhase(ctx)
if err != nil {
t.Fatal(err)
}
return s.st
}
// TestScanOneFilesystemKeepsOtherFilesystemRecords checks that with -x a
// record below a directory the walk does not enter, because it is on
// another filesystem than its operand, counts as outside the operands:
// it is not deleted, and its size joins the size census. The record of
// a gone file that is not below that directory is still deleted, even
// when its path begins with the directory's name. The operand is given
// a device that no directory here is on, so no second real filesystem
// is needed.
func TestScanOneFilesystemKeepsOtherFilesystemRecords(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
// A scan without -x records the file below mnt, and mnt2, which is
// beside mnt, not below it.
mounted := writeFile(t, dir, "mnt/f", pattern(1, 100))
gone := writeFile(t, dir, "mnt2", pattern(3, 50))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
// The same size as mnt/f, so it is hashed only if mnt/f's size is
// in the census.
added := writeFile(t, dir, "added", pattern(2, 100))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
syncOneFilesystem(t, db, []string{dir}, map[string]uint64{dir: dev + 1})
recs := dbRecords(t, db)
recordByPath(t, recs, mounted)
if recordByPath(t, recs, added).head == "" {
t.Errorf("%s not hashed: the census lacks the size of %s",
added, mounted)
}
if slices.Contains(recordPaths(recs), gone) {
t.Errorf("record for %s kept: the file is gone and not below mnt",
gone)
}
}
// TestScanOneFilesystemWalksNestedOperand checks that with -x an
// operand under another one, on a different filesystem than it, is
// walked on its own: a file added below it is recorded, and the record
// of a file gone from below it is deleted. A record below a directory
// that its own walk does not enter is still kept. Each operand is given
// a device that no directory here is on, so neither walk enters a
// subdirectory.
func TestScanOneFilesystemWalksNestedOperand(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
// A scan without -x records both files below mnt.
gone := writeFile(t, dir, "mnt/gone", pattern(1, 10))
kept := writeFile(t, dir, "mnt/sub/kept", pattern(2, 20))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
added := writeFile(t, dir, "mnt/added", pattern(3, 30))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
mnt := filepath.Join(dir, "mnt")
syncOneFilesystem(t, db, []string{dir, mnt},
map[string]uint64{dir: dev + 1, mnt: dev + 2})
paths := recordPaths(dbRecords(t, db))
if !slices.Contains(paths, added) {
t.Errorf("no record for %s: the walk did not reach %s", added, mnt)
}
if slices.Contains(paths, gone) {
t.Errorf("record for %s kept: the file is gone", gone)
}
if !slices.Contains(paths, kept) {
t.Errorf("record for %s deleted: the walk of %s did not enter %s",
kept, mnt, filepath.Dir(kept))
}
}
// TestScanOneFilesystemWalksOperandBelowOtherFilesystem checks that
// with -x an operand on the same filesystem as the operand covering it
// is walked on its own when a directory between them is on another
// filesystem, since the walk of the covering operand does not enter
// that directory: a file added below the nested operand is recorded,
// and the record of a file gone from below it is deleted. A record
// below that directory but outside the nested operand is still kept.
// Both operands are given one device that no directory here is on,
// while mnt keeps its own.
func TestScanOneFilesystemWalksOperandBelowOtherFilesystem(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
// A scan without -x records both files below mnt.
gone := writeFile(t, dir, "mnt/back/gone", pattern(1, 10))
kept := writeFile(t, dir, "mnt/kept", pattern(2, 20))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
added := writeFile(t, dir, "mnt/back/added", pattern(3, 30))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
mnt := filepath.Join(dir, "mnt")
back := filepath.Join(mnt, "back")
syncOneFilesystem(t, db, []string{dir, back},
map[string]uint64{dir: dev + 1, mnt: dev, back: dev + 1})
paths := recordPaths(dbRecords(t, db))
if !slices.Contains(paths, added) {
t.Errorf("no record for %s: the walk did not reach %s", added, back)
}
if slices.Contains(paths, gone) {
t.Errorf("record for %s kept: the file is gone", gone)
}
if !slices.Contains(paths, kept) {
t.Errorf("record for %s deleted: the walk of %s did not enter %s",
kept, dir, mnt)
}
}
// TestScanOneFilesystemRecordsNestedFileOnce checks that with -x a
// regular-file operand under another operand is dropped when the file
// reports a different device than that operand but its directory does
// not, as a file bind-mounted into the tree does: the walk of the other
// operand enters the directory and reaches the file, so the file is
// recorded once.
func TestScanOneFilesystemRecordsNestedFileOnce(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
file := writeFile(t, dir, "sub/f", pattern(1, 10))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
st := syncOneFilesystem(t, db, []string{dir, file}, map[string]uint64{
dir: dev, filepath.Dir(file): dev, file: dev + 1,
})
if st.added != 1 {
t.Errorf("%s recorded %d times, want once", file, st.added)
}
}
// TestScanOneFilesystemWalksNestedFileOperand checks that with -x a
// regular-file operand under another operand is kept when its directory
// is on a different filesystem than that operand: the walk of the other
// operand does not enter the directory, so the file is recorded once,
// as its own operand. The other operand is given a device that no
// directory here is on.
func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
file := writeFile(t, dir, "sub/f", pattern(1, 10))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
st := syncOneFilesystem(t, db, []string{dir, file}, map[string]uint64{
dir: dev + 1, filepath.Dir(file): dev,
})
if st.added != 1 {
t.Errorf("%s recorded %d times, want once", file, st.added)
}
}
// buildSmokeTree recreates the README smoke-test filesystem layout // buildSmokeTree recreates the README smoke-test filesystem layout
// with deterministic content and returns the tree root. // with deterministic content and returns the tree root.
func buildSmokeTree(t *testing.T) string { func buildSmokeTree(t *testing.T) string {
@@ -1285,11 +1643,172 @@ func TestSyncScanMtimeBump(t *testing.T) {
t.Fatalf("mtime-bump stats = %+v, want 1 updated", st) t.Fatalf("mtime-bump stats = %+v, want 1 updated", st)
} }
if r := recordByPath(t, dbRecords(t, db), a); r.mtime != future.Unix() { if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(future) {
t.Fatalf("mtime = %d, want %d", r.mtime, future.Unix()) t.Fatalf("mtime = %v, want %v", r.mtime, future)
} }
} }
// assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
data []byte,
) {
t.Helper()
r := recordByPath(t, dbRecords(t, db), path)
if want := hexSum(data); r.head != want || r.tail != want ||
r.content != want {
t.Fatalf("head, tail, content = %q, %q, %q, want %q for each",
r.head, r.tail, r.content, want)
}
}
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
// size with an mtime later in the same second as the recorded one: the
// next scan must notice the change and re-hash the file.
func TestSyncScanSameSecondRewrite(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
writeFile(t, dir, "b.bin", pattern(2, 500))
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(a, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(a, later, later)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
// TestSyncScanOperandSameSecondRewrite is TestSyncScanSameSecondRewrite
// for files given to scan as operands, which scan stats without reading
// their directory.
func TestSyncScanOperandSameSecondRewrite(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
b := writeFile(t, dir, "b.bin", pattern(2, 500))
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(a, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, a, b)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(a, later, later)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, a, b)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
// TestSyncScanRewriteAfter2262 runs assertLateRewriteRehashed with an
// mtime after 2262, a time too late to count in nanoseconds in an int64.
func TestSyncScanRewriteAfter2262(t *testing.T) {
t.Parallel()
assertLateRewriteRehashed(t, time.Date(2300, 1, 2, 3, 4, 5, 0, time.UTC))
}
// TestSyncScanRewritePastTimeLimit runs assertLateRewriteRehashed with an
// mtime one second past the latest a time.Time holds without wrapping it
// to a time far in the past.
func TestSyncScanRewritePastTimeLimit(t *testing.T) {
t.Parallel()
assertLateRewriteRehashed(t, time.Unix(9223371974719179008, 0))
}
// assertLateRewriteRehashed scans a directory, rewrites a file in it in
// place at the same size, sets its mtime to late, and fails unless the
// next scan re-hashes the file. It skips where late does not fit the
// platform's timespec or the filesystem does not store it.
func assertLateRewriteRehashed(t *testing.T, late time.Time) {
t.Helper()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
writeFile(t, dir, "b.bin", pattern(2, 500))
syncTree(t, db, dir)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
// os.Chtimes cannot set such a time: it converts through UnixNano.
ts, err := unix.TimeToTimespec(late)
if err != nil {
t.Skipf("an mtime %d seconds after 1970 does not fit this platform's "+
"timespec: %v", late.Unix(), err)
}
err = unix.UtimesNano(a, []unix.Timespec{ts, ts})
if err != nil {
t.Fatal(err)
}
fi, err := os.Lstat(a)
if err != nil {
t.Fatal(err)
}
if fi.ModTime().Unix() != late.Unix() {
t.Skipf("the filesystem stored the mtime as %d seconds after 1970, "+
"not %d", fi.ModTime().Unix(), late.Unix())
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
func TestSyncScanAddRemove(t *testing.T) { func TestSyncScanAddRemove(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1335,9 +1854,7 @@ func TestSyncScanSizeChange(t *testing.T) {
writeFile(t, dir, "f", pattern(1, 200)) writeFile(t, dir, "f", pattern(1, 200))
mt := time.Unix(old.mtime, 0) err := os.Chtimes(p, old.mtime, old.mtime)
err := os.Chtimes(p, mt, mt)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -1577,6 +2094,54 @@ func TestScanHardlinkRunFailsTogether(t *testing.T) {
} }
} }
// TestHashPhaseSkipsFileReplacedByFIFO replaces a walked regular file
// with a FIFO before the hash phase reads it. Opening a FIFO that has
// no writer waits for one, so a blocking open would hang the phase. The
// path must be warned about once and skipped, with no record written.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
p := writeFile(t, dir, "a.bin", pattern(1, 300))
walked, _ := collectWalk(t, []string{dir}, false, 1)
err := os.Remove(p)
if err != nil {
t.Fatal(err)
}
err = unix.Mkfifo(p, 0o600)
if err != nil {
t.Fatal(err)
}
s := &scanState{db: openTestDB(t), toHash: walked}
err = s.hashPhase(t.Context(), 1)
if err != nil {
t.Fatalf("hashPhase = %v, want nil", err)
}
if s.st.skipped != 1 || len(s.batch) != 0 {
t.Errorf("skipped = %d, records = %d, want 1 skipped and none",
s.st.skipped, len(s.batch))
}
warning := "hash " + p + ": not a regular file\n"
if got := stderr(); strings.Count(got, warning) != 1 {
t.Errorf("stderr = %q, want %q once", got, warning)
}
// The content phase opens files the same way.
_, _, content, err := hashContentOnly(p, headTailMin)
if content != "" || !errors.Is(err, errNotRegular) {
t.Errorf("hashContentOnly on a FIFO = %q, %v; want no hash and %v",
content, err, errNotRegular)
}
}
// injectedWriteFailure is the message the injected database trigger // injectedWriteFailure is the message the injected database trigger
// aborts with, so the test can recognize its own failure in the error // aborts with, so the test can recognize its own failure in the error
// the scan reports. // the scan reports.
@@ -1770,14 +2335,43 @@ func TestHashRuns(t *testing.T) {
func TestPruneRoots(t *testing.T) { func TestPruneRoots(t *testing.T) {
t.Parallel() t.Parallel()
const pqRoot = "/p/q"
// /p/q is on another filesystem than /p; /p/q/r is on the same one as
// /p/q, /p/s as /p, and /p/t has no known device.
devices := map[string]uint64{"/p": 1, pqRoot: 2, "/p/q/r": 2, "/p/s": 1}
device := func(path string) (uint64, bool) {
dev, ok := devices[path]
return dev, ok
}
// Duplicates and operands under other operands are dropped; /cc is // Duplicates and operands under other operands are dropped; /cc is
// not under /c (sibling with a shared prefix). // not under /c (sibling with a shared prefix).
got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"}) got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
false, device)
want := []string{"/a", "/c", "/cc"} want := []string{"/a", "/c", "/cc"}
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Fatalf("pruneRoots = %q, want %q", got, want) t.Fatalf("pruneRoots = %q, want %q", got, want)
} }
nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
got = pruneRoots(nested, false, device)
want = []string{"/p"}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots without -x = %q, want %q", got, want)
}
// With -x, /p/q, on another filesystem than /p, is kept as well.
got = pruneRoots(nested, true, device)
want = []string{"/p", pqRoot}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
}
} }
func TestReportsNeverTouchFilesystem(t *testing.T) { func TestReportsNeverTouchFilesystem(t *testing.T) {