diff --git a/README.md b/README.md index 4aaca1a..6a4b5c3 100644 --- a/README.md +++ b/README.md @@ -3,22 +3,22 @@ ## Description `sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that -quickly identifies _candidate_ duplicate files — and, ultimately, entire -duplicate directory trees — across very large filesystems without reading every -byte of every file. Files are considered duplicates when their sizes are equal -and they agree on a short ladder of hashes. A file under 10 MiB is hashed in -full and compared directly. A larger file is gated first on the SHA-256 of its -first 64 KiB and of its last 64 KiB, and only when its size and both of those -match another file's is it read for a content hash to compare — the SHA-256 of -the whole file when it is under 50 MiB, or of gigabyte-spaced 1 MiB samples when -it is 50 MiB or larger. Below 50 MiB the content hash is proof of identical -content; at or above 50 MiB it is a strong candidate signal rather than proof, -because the gaps between samples are never read. The intended use is finding -duplicate downloads and duplicated directory trees on multi-terabyte ZFS servers -where reading every byte of every file is prohibitively expensive. `scan` -maintains a persistent SQLite database of file signatures that survives between -runs, so it can be run from cron and the reports can be generated at any time -from the most recent scan. +quickly finds _candidate_ duplicate files, and ultimately whole duplicate +directory trees, on very large filesystems. It does this without reading every +byte of every file. + +Two files are duplicates when they have the same size and the same three SHA-256 +hashes. Two of the hashes cover the first and the last 64 KiB of the file. The +third, the content hash, covers the whole file when it is under 100 MiB, so a +match proves identical content. At 100 MiB and above it covers only 1 MiB at +each gigabyte, so such a file is never read in full. A match there is a strong +candidate rather than proof. + +The intended use is finding duplicate downloads and duplicated directory trees +on multi-terabyte ZFS servers, where reading every byte of every file is +prohibitively expensive. `scan` keeps a persistent SQLite database of file +signatures between runs, so it can run from cron. The reports can be generated +at any time from the most recent scan. This README is the complete and authoritative specification. @@ -34,13 +34,13 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite" `scan` walks one or more filesystem trees and maintains one database record per regular file (path, size, mtime, head hash, tail hash, content hash). The -database persists between runs; a rescan only hashes files that are new or -changed, or that may have gained a duplicate since the last scan, and removes -records for files that no longer exist. `report` reads the database and prints -the file-level duplicates report. `trees` reads the same database and prints the -duplicate-tree report. `info` prints the database's details: its path, the sizes -of its files and the number of its records. A missing/invalid subcommand — or a -`scan` invocation with no `PATH` operand — prints a usage message and exits 2. +database persists between runs. A rescan hashes only the files that are new or +changed, and removes the records of files that no longer exist. `report` reads +the database and prints the file-level duplicates report. `trees` reads the same +database and prints the duplicate-tree report. `info` prints the database's +details: its path, the sizes of its files and the number of its records. A +missing/invalid subcommand — or a `scan` invocation with no `PATH` operand — +prints a usage message and exits 2. The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan` @@ -122,14 +122,19 @@ As a file `/etc/cron.d/sfdupes`: the first scan. To keep it elsewhere, set `SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line. - `scan` writes nothing to stdout. Its stderr, appended here to - `/var/log/sfdupes.log`, holds a first line naming the database (see - "Database"), plain progress lines naming each phase and its place among the - five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it - runs, and one as it ends), a warning for each path it skips, a line for each - default exclude its walk does not enter and for each path it leaves out - because it is not on a persistent filesystem (see "Rules for the walk"), and - the summary line (see "Progress" and "`scan` mode"). The log grows with every - scan; rotate it like any other. + `/var/log/sfdupes.log`, holds these lines: + - a first line naming the database (see "Database"); + - plain progress lines naming each phase and its place among the four, as + `[2/4] walk`: one as the phase starts, one every 5 seconds while it runs, + and one as it ends; + - a warning for each path it skips; + - a line for each default exclude its walk does not enter, and for each path + it leaves out because it is not on a persistent filesystem (see "Rules for + the walk"); + - the summary line (see "Progress" and "`scan` mode"). + + The log grows with every scan; rotate it like any other. + - Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the reason among the last lines of the log; `tail` prints them, and cron mails @@ -153,7 +158,7 @@ A row is a candidate, not proof: - The reports read only the database, so they show the files as of the last scan; a file may have changed or gone since. -- A file of 50 MiB or more is compared only on samples of its content (see +- A file of 100 MiB or more is compared only on samples of its content (see "Duplicate detection"). - Paths that are hard links to one file are listed as duplicates, but they share their data, so removing one frees nothing. @@ -188,22 +193,23 @@ or a symlink, which `trees` does not see. ## Rationale -Duplicate finders that hash entire files do not scale to the target environment: -~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under -resilver). sfdupes spends disk I/O only on files whose size at least one other -file shares, since a size-unique file cannot be a duplicate. Of those, a file -under 10 MiB is read in full; a larger one has its cheap end windows read first, -and is read for a content hash only when its size and both end windows match -another file's — the whole file below 50 MiB, but only gigabyte-spaced samples -at or above 50 MiB, so the largest files are never read in full. This keeps a -full-filesystem sweep tractable, and the signatures are kept in a persistent -database, so the expensive filesystem pass is incremental: a rescan re-hashes -only files whose recorded mtime or size changed, plus — for its content hash — a -file of 10 MiB or more whose size and end windows have come to match another -file's. All analysis happens offline from the database alone. The end goal is -not individual files but whole duplicated trees — duplicate extractions, -duplicate downloads, copied project trees — which an operator can consider -removing as a unit. +Duplicate finders that hash every file in full do not scale to the target +environment. It holds about 10 million files and 150 TB, on disks that may be +slow or busy, such as a ZFS pool under resilver. + +sfdupes reads a file under 100 MiB in full, and a larger one only at its two +ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it +makes the comparison exact. A full read of every large file would take +unreasonably long. Sampling cuts a 150 GiB file to about 150 MiB of reads, at +the price of a comparison that is not exact. + +The hashes are kept in a persistent database, so the expensive filesystem pass +is incremental. A rescan reads only the files that are new or whose size or +mtime changed. All analysis then works from the database alone. + +The end goal is not individual files but whole duplicated trees, such as a +download extracted twice. An operator can consider removing such a tree as a +unit. ## Design @@ -215,14 +221,10 @@ Goals, in order: trees), so the operator can consider removing an entire subtree at once. File-level duplicate detection is the foundation; tree-level detection is built on top of it. -2. **Spend I/O in proportion to duplicate likelihood.** Only files whose size - at least one other file shares are read at all — a size-unique file cannot - be a duplicate. Those are compared by the ladder in "Duplicate detection" - below: a file under 10 MiB is hashed in full, while a larger file is gated - on cheap 64 KiB end windows first, and gets a content hash only when its - size and both end windows match another file's. That hash reads the whole - file below 50 MiB but only gigabyte-spaced 1 MiB samples at or above it, so - the very largest files are still never read in full. Scale target: tens of +2. **Bound the I/O per file.** A file is read only when it is new or has + changed. A file under 100 MiB is then read in full, and a larger one only at + its ends and at 1 MiB per gigabyte (see "Duplicate detection" below). + Therefore no file of 100 MiB or more is read in full. Scale target: tens of millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool under resilver). Holding one small record (path, size, mtime) per file in memory during a scan is acceptable; holding every file's hashes is not (they @@ -235,12 +237,10 @@ Goals, in order: being read, so its memory grows with the number of directories and with the size of the largest directory. 3. **Scan incrementally, analyze offline.** The expensive filesystem scan - maintains a persistent database; an unchanged file is never read again on a - rescan, except to compute its content hash once a file of 10 MiB or more - comes to match another on size and both end windows. All analysis (`report`, - `trees`) works from the database alone and must never touch the scanned - filesystem again. `scan` is designed to be cronned; the reports run at any - time against the last completed scan. + maintains a persistent database, and an unchanged file is never read again + on a rescan. All analysis (`report`, `trees`) works from the database alone + and must never touch the scanned filesystem again. `scan` is designed to be + cronned; the reports run at any time against the last completed scan. 4. **Clean stream separation.** Everything on stdout is machine-readable data. All progress, warnings, summaries, and help and usage text go to stderr. Never mix them. @@ -272,7 +272,7 @@ Four subcommands, all implemented: from the database records, compute a Merkle-style digest per directory, and report maximal groups of identical trees. 4. `info` — the database's details: its path, the sizes of its files, its - schema version and the counts of its records. + schema version, the number of its records and their total size. ``` sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... @@ -368,9 +368,7 @@ All four subcommands operate on a single SQLite database file: - While it loads the existing records (`load`) and walks: no records. A first scan's database holds none, however long its walk has run. - During the hash phase: the first records, committed 10,000 at a time. - - In the update phase: the rest of the hashed records, the records for - size-unique files, and the deletions. - - In the content phase: content hashes, again committed 10,000 at a time. + - In the update phase: the rest of the hashed records, and the deletions. - At the end, as it closes the database: the `-wal` and `-shm` files go away, unless a report still has the database open (see above). - `report`, `trees` and `info` open the database read-only and need only read @@ -390,9 +388,9 @@ All four subcommands operate on a single SQLite database file: size INTEGER NOT NULL, -- bytes, from lstat mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999 - head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB - tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB - content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples + head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB + tail BLOB NOT NULL, -- raw 32-byte SHA-256 of the last 64 KiB + content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples ) WITHOUT ROWID; CREATE INDEX files_signature ON files (size, head, tail, content); ``` @@ -403,57 +401,46 @@ All four subcommands operate on a single SQLite database file: nanoseconds past that second. Split this way they hold any mtime a filesystem can record, one before 1678 or after 2262 included, which a single 64-bit count of nanoseconds cannot. They are used only for change - detection and are not part of the duplicate key. For a file under 10 MiB - `head`, `tail`, and `content` all hold the whole-file hash (that range is - hashed in full, with no end windows); for a larger file `head` and `tail` - hold the first- and last-64 KiB hashes and `content` the whole-file or - sampled hash. All three are empty strings when the file has never been - hashed because its size was unique as of the last scan that covered it. For - a file of 10 MiB or more, `content` stays empty until the content phase of a - scan (see "`scan` mode" below) has read the file. A record with an empty - `content` is never part of a duplicate group, though it still defines the - file for tree reconstruction. The `files_signature` index lets `report` read - the records in signature order without sorting the table. + detection and are not part of the duplicate key. Every record holds the + three hashes that "Duplicate detection" below describes. The + `files_signature` index lets `report` read the records in signature order + without sorting the table. ### Duplicate detection -Two files are duplicates only when they agree on every rung of this ladder; a -mismatch at any rung means they are not duplicates. `scan` stores each file's -hashes, and `report` and `trees` group files by the whole signature — size, -`head`, `tail`, and `content` — so the grouping is exactly this ladder applied -across everything scanned into the database, even across separate scans. +Two files are duplicates when they have the same size and the same three hashes. +`scan` computes the hashes and stores them in the database. `report` and `trees` +group the files whose size and hashes all match, whichever scan recorded them. -1. **Size.** Files of different sizes are never compared. Only files whose size - at least one other file shares are hashed at all. -2. **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed in full - and compared directly, with no separate end-window step — small files are - cheap to read to the last byte, and doing so makes the comparison exact. - `head`, `tail`, and `content` all hold this whole-file SHA-256, so such a - file's signature is decided entirely by its size and its content. -3. **10 MiB and above: head and tail.** For a larger file, the SHA-256 of the - first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a cheap gate that - eliminates most same-size pairs before any bulk reading: the content hash of - the next two rungs is computed only for a file whose size, `head`, and - `tail` match another file's, whether that file is scanned in the same run or - stored by an earlier scan. A stored file that first gains such a match in a - later scan gets its content hash then; until it has one, its `content` is - empty and it is not a duplicate. At 10 MiB and above the two windows never - overlap. -4. **10 MiB and above, content below 50 MiB.** The SHA-256 of the entire file. - Agreement here is proof of identical content (barring a SHA-256 collision). -5. **10 MiB and above, content 50 MiB and above.** A sampled SHA-256: the 1 MiB - window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … while inside the - file, the final window truncated at end of file) is fed, in order, into one - hash. This is **deliberately probabilistic** — the gaps between samples are - never read, so two large files that agree on every sample are reported as - duplicates without being read in full. It is the price of never reading a - 150 GB file end to end. Because size is already part of the signature, only - equal-size files reach this rung, so their sample boundaries always align. +`scan` gives every file it records three SHA-256 hashes: -`head`, `tail`, and `content` are one column each. A file below 10 MiB and one -at or above it never share a size, and neither do a file below 50 MiB and one at -or above it, so a stored value is never ambiguous between the whole-file, -end-window, and sampled forms. +- `head`, the hash of the file's first 64 KiB. +- `tail`, the hash of the file's last 64 KiB. +- `content`, the content hash described below. + +In a file of 64 KiB or less, the first and the last 64 KiB are both the whole +file, so all three hashes are the hash of the whole file. + +The content hash depends on the file's size: + +- Under 100 MiB, it is the SHA-256 of the whole file. Agreement on it is proof + of identical content, barring a SHA-256 collision. +- At 100 MiB and above, it is the SHA-256 of samples, so the file is never read + in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2 + GiB and so on. The samples are fed into one hash in order. A sample that would + run past the end of the file stops there. + +The sampled hash is **deliberately probabilistic**. Bytes outside the samples +and outside the first and last 64 KiB are never read. Two large files that +differ only in such bytes are reported as duplicates. That is the price of never +reading a 150 GB file end to end. + +Only files of the same size are compared, so their samples always lie at the +same offsets. A file under 100 MiB and one of 100 MiB or more never share a +size, so a whole-file hash is never compared with a sampled one. + +`scan` reads a file only to hash it, and only when the file is new or has +changed since its record was written (see "`scan` mode"). ### `scan` mode @@ -474,40 +461,36 @@ exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with `-x`, is on a different filesystem than that operand, because that operand's walk does not enter such a directory and so does not reach it. -An operand that is a symlink (never followed, not even as an operand), socket, -FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints -a one-line warning naming the path and what it is, counts it as skipped, and -drops it from the scanned operands before reading the database. Another operand -beneath it is still scanned. The records stored beneath it are not deleted: they -are treated like any other record outside the scanned operands, including the -content-phase exception below. If it lies under another operand, they are under -that operand instead, and are treated like any other record there. This is not -an error: a scan whose every operand is dropped walks nothing and exits 0. An -operand that is not on a persistent filesystem (see "Rules for the walk") is -dropped the same way, except that the line `scan` prints for it is not a -warning, and it is not counted as skipped. +An operand is not scanned when it is: + +- a symlink, which `scan` never follows, not even as an operand; +- a socket, FIFO or device node; +- at or inside a directory named `.zfs`. + +An operand reached through a symlink into a `.zfs` directory is not scanned +either. + +This holds even when the operand lies under another operand. `scan` prints a +one-line warning naming the path and what it is, counts it as skipped, and drops +it from the scanned operands before reading the database. An operand beneath a +symlink operand is still scanned. The records stored beneath the dropped operand +are not deleted: they are treated like any other record outside the scanned +operands. If it lies under another operand, they are under that operand instead, +and are treated like any other record there. This is not an error: a scan whose +every operand is dropped walks nothing and exits 0. An operand that is not on a +persistent filesystem (see "Rules for the walk") is dropped the same way. The +line `scan` prints for it is not a warning, though, and it is not counted as +skipped. `scan` synchronizes the database with the filesystem state under the scanned operands: -- Only a file whose size at least one other file shares is ever read: a - size-unique file cannot be a duplicate, so it is recorded without hashes - (`head`, `tail`, and `content` empty). The size census covers every file - walked this scan plus every database record outside the scanned operands, so a - possible duplicate of a separately scanned tree is still recognized. -- A file not yet in the database is inserted: hashed when its size is shared, - without hashes otherwise. +- A file not yet in the database is hashed and inserted. - A file already in the database is **skipped without reading its contents** when its lstat size equals the recorded size and its lstat mtime is not newer - than the recorded mtime. This is what makes a daily rescan cheap. Exception: - an unchanged file whose record lacks hashes is hashed — and its record updated - — once its size becomes shared, so hashing deferred by size-uniqueness happens - as soon as it could matter. Likewise, an unchanged file of 10 MiB or more - whose record has no `content` hash is read for one by the content phase below - once its size, `head`, and `tail` match another record's. -- A file whose mtime is newer than recorded, or whose size differs, is processed - as if new: re-hashed, or recorded without hashes, per the shared-size rule. - Change detection compares the mtime to the nanosecond, as finely as the + than the recorded mtime. This is what makes a daily rescan cheap. +- A file whose mtime is newer than recorded, or whose size differs, is hashed + again. Change detection compares the mtime to the nanosecond, as finely as the filesystem records it, so a same-size rewrite counts as a change whenever the filesystem gives it a later mtime than recorded, even within the same second. - A database record whose path lies under one of the scanned operands but was @@ -515,31 +498,26 @@ operands: deleted files. It also removes records for paths that failed to stat or hash this run: the database only ever contains signatures verified by the most recent scan that covered them (a subsequent successful scan re-adds such - files). A failure in the content phase below removes nothing: the record is - left as it is. + files). - Database records outside the scanned operands are untouched, so disjoint trees - can be scanned on different schedules into the same database. The one - exception is the content phase below: a stored file of 10 MiB or more without - a `content` hash is read for one, wherever it lies, once its size, `head`, and - `tail` match another record's. If that file is gone, has changed since its - record was written, or is not on a persistent filesystem, it is not read and - the record is left as it is. -- A record at or below a path the walk left out, because that path is a default - exclude, is not on a persistent filesystem, or, with `-x`, is a directory on - another filesystem than its operand, counts as outside the scanned operands: - it is not deleted, and its size joins the size census. A filesystem mounted - under an operand can therefore be scanned on its own schedule into the same - database. Records below an operand at or inside such a path are not kept this - way, since that operand's own walk verifies them. + can be scanned on different schedules into the same database. +- A record at or below a path the walk left out counts as outside the scanned + operands, so it is not deleted. The walk leaves out a path that: + - is a default exclude; + - is not on a persistent filesystem; + - with `-x`, is a directory on another filesystem than its operand. -`scan` runs **five sequential phases over the whole scan**. Parallelism lives -inside the walk, hash and content phases; batched database writes begin during -the hash phase: + A filesystem mounted under an operand can therefore be scanned on its own + schedule into the same database. Records below an operand at or inside such + a path are not kept this way, since that operand's own walk verifies them. + +`scan` runs **four sequential phases over the whole scan**. Parallelism lives +inside the walk and hash phases, and batched database writes begin during the +hash phase: 1. **load** — read the existing records. Each record under the operands is kept - in memory, by path, with its size, its mtime and whether it has hashes, for - the walk to compare each file with; the size of every other record joins the - size census. + in memory, by path, with its size and its mtime, for the walk to compare + each file with. 2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently with the walk worker pool: every operand seeds the shared queue, and each worker reads one directory at a time, handing discovered subdirectories back @@ -547,49 +525,23 @@ the hash phase: (while the directory's metadata is still hot). Sequential directory enumeration is metadata-latency-bound and takes hours at tens of millions of files; per-directory parallelism is what makes the walk tractable on large - or busy pools. The walk builds the size census and resolves unchanged - already-hashed files on the fly; every other file is carried to the hash - phase as a (path, size, mtime) record. -3. **hash** — with the census complete, each carried file's size decides its - fate. Size-unique files are never read: new or changed ones are recorded - without hashes in the update phase, unchanged unhashed ones simply keep - their records. Every file with a shared size is hashed by the worker pool as - described in "Duplicate detection" above: a file under 10 MiB in full, which - gives its `head`, `tail`, and `content` alike, and a larger file only in its - end windows, which give its `head` and `tail`; its content hash is left to - the content phase. Zero-length files have constant hashes and are never - opened. Files are hashed in **inode order** (minimizing seeks on spinning - disks), and paths that are hard links to the same inode are **read once**, - all sharing the one result — a hard-link backup farm costs one read per - inode, not per path. The phase total counts actual reads, so progress and - ETA are meaningful. Completed records are committed in batched transactions - **while hashing runs**, so a scan interrupted after hours keeps everything - hashed so far and the next scan resumes cheaply, skipping records already - written. -4. **update** — commit the final partial batch, the hash-less records for - size-unique new and changed files, and the deletions for records the scan - did not verify (vanished files, plus paths that failed to stat or hash). -5. **content** — find every record of 10 MiB or more without a `content` hash - whose size, `head`, and `tail` equal another record's, anywhere in the - database: records from this scan and records stored by earlier scans, inside - or outside the scanned operands. SQLite finds them, so only the records to - be read are kept in memory, never every file's hashes. Every record sharing - their size, `head`, and `tail`, including one that already has a `content` - hash, has its file checked with `lstat` first. A file that is gone, is no - longer a regular file, has changed (a different size, or an mtime newer than - recorded), or is not on a persistent filesystem (see "Rules for the walk") - keeps its record as it is and does not count as a match for the others. Any - other `lstat` error is warned about and counted as skipped, with the same - result. If such a record has no `content` hash, it stays out of duplicate - groups; if it has one, it is still reported until a scan covering its own - tree updates or removes it. The files that pass and have no `content` hash - are read only if at least two of those records pass, so a file whose only - matches are stale costs no read; a file that already has a `content` hash is - never read again. They are read by a worker pool as in the hash phase, in - inode order and once per inode, and their content hashes are committed in - batches. A failed read is warned about and counted as skipped; its record - keeps an empty `content`, so it is not a duplicate, and a later scan tries - again. + or busy pools. The walk compares each file with its record as it goes, and + an unchanged file needs nothing more. Every new or changed file is carried + to the hash phase as a (path, size, mtime) record. +3. **hash** — the worker pool hashes every carried file as described in + "Duplicate detection" above, which gives the file its `head`, `tail` and + `content`. Zero-length files have constant hashes and are never opened. + Files are hashed in **inode order**, which minimizes seeks on spinning + disks. Paths that are hard links to the same inode are **read once**, and + all share the one result. A hard-link backup farm therefore costs one read + per inode, not per path. The phase total counts actual reads, so its + progress and its time remaining are meaningful. Completed records are + committed in batched transactions **while hashing runs**. A scan interrupted + after hours therefore keeps everything hashed so far, and the next scan + resumes cheaply, skipping records already written. +4. **update** — commit the final partial batch of hashed records. Then delete + the records the scan did not verify: those of vanished files, and of paths + that failed to stat or hash. Rules for the walk: @@ -612,10 +564,10 @@ Rules for the walk: - Filesystem boundaries are crossed by default. With `-x` (long form `--one-file-system`, following the GNU `du`/`rsync` convention), never descend into a directory on a different filesystem than its `PATH` operand; each - operand is bounded by its own filesystem. The records below such a directory, - unless they lie below an operand at or inside that directory, count as outside - the scanned operands, as described in "`scan` mode" above: they are not - deleted, and their sizes join the size census. + operand is bounded by its own filesystem. The records below such a directory + count as outside the scanned operands, as described in "`scan` mode" above: + they are not deleted. That does not hold below an operand at or inside that + directory. - Never record anything that is not on a persistent filesystem. A filesystem is not persistent when the kernel makes it up or keeps it in memory: one of type `proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`, @@ -632,21 +584,19 @@ Rules for the walk: `scan` cannot list them, it fails (exit 1). The walk then never enters a directory, and never records a file, on one of those devices, however it reaches it: as an operand, below one, or through a bind mount, which keeps the - device ID of what it binds. The content phase never reads a file on one of - them either. No flag turns this off. For each operand, directory or file it - leaves out this way, `scan` prints a line on stderr naming it and the type of - its filesystem, as it prints a warning, so it never lands inside a progress - display: + device ID of what it binds. No flag turns this off. For each operand, + directory or file it leaves out this way, `scan` prints a line on stderr + naming it and the type of its filesystem. It prints that line as it prints a + warning, so the line never lands inside a progress display: ``` scan: skipped /dev/shm (tmpfs, not a persistent filesystem) ``` - That line is not a warning, and the path is not counted as skipped. Records - that an earlier scan left at or below it, unless they lie below an operand - at or inside it, count as outside the scanned operands, as described in - "`scan` mode" above: they are not deleted, and their sizes join the size - census. + That line is not a warning, and the path is not counted as skipped. An + earlier scan's records at or below it count as outside the scanned operands, + as described in "`scan` mode" above: they are not deleted. That does not + hold below an operand at or inside it. - Below an operand, never descend into a default exclude: a path that holds the kernel's view of the running system, or temporary files, so that @@ -678,32 +628,35 @@ Rules for the walk: That line is not a warning, and the directory is not counted as skipped. A default exclude that is not on a persistent filesystem, as `/proc` is, gets - the line for its filesystem instead, and one that is itself an operand gets - no line, since it is walked as that operand. The records below a default - exclude the walk does not enter, unless they lie below an operand at or - inside it, count as outside the scanned operands, as described in "`scan` - mode" above: they are not deleted, and their sizes join the size census. + the line for its filesystem instead. One that is itself an operand gets no + line, since it is walked as that operand. When the walk does not enter a + default exclude, the records below it count as outside the scanned operands, + as described in "`scan` mode" above: they are not deleted. That does not + hold below an operand at or inside it. -- On any per-path error (permission denied, file vanished between passes, - unreadable): print a one-line warning to stderr, skip the path, and continue. - Per-file errors never abort the run; the final summary reports how many were - skipped. As specified above, a skipped path that has a database record from an - earlier scan loses that record, unless it failed only in the content phase, or - is an operand dropped before the database was read whose record counts as - outside the scanned operands; an unreadable directory subtree likewise loses - its records (accepted: the database mirrors what the latest scan could - actually verify). +- On a per-path error, such as permission denied or an unreadable file: print a + one-line warning to stderr, skip the path, and continue. Per-file errors never + abort the run; the final summary reports how many were skipped. As specified + above, a skipped path that has a database record from an earlier scan loses + that record. An unreadable directory subtree likewise loses its records. This + is accepted: the database mirrors what the latest scan could actually verify. + The exception is an operand dropped before the database was read, whose record + counts as outside the scanned operands. +- A file or directory removed while the scan runs is not an error. The scan + prints no line for it and does not count it as skipped. A path gone when the + walk reaches it gets no record. A file gone when the hash phase opens it gets + none either. The records an earlier scan left at or below such a path are + deleted. -Concurrency: the walk phase (which also stats files), the hash phase, and the -content phase each use a worker pool of `--workers` workers (default -`runtime.NumCPU()`); the walk parallelizes across directories, hashing across -files. `--workers` must be at least 1: a smaller value is a usage error, -reported in one line on stderr with exit 2 before anything is scanned. All three -phases are seek-bound on spinning disks, so raising `--workers` well past the -core count can help on pools with many spindles. The main goroutine owns -partitioning, database writes, and the progress counts, which each progress -display draws from a goroutine of its own; progress display must never block the -workers. +Concurrency: the walk phase, which also stats files, and the hash phase each use +a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk +parallelizes across directories, and hashing across files. `--workers` must be +at least 1: a smaller value is a usage error, reported in one line on stderr +with exit 2 before anything is scanned. Both phases are seek-bound on spinning +disks, so raising `--workers` well past the core count can help on pools with +many spindles. The main goroutine owns the database writes and the progress +counts. Each progress display draws the counts from a goroutine of its own, and +must never block the workers. `scan` writes nothing to stdout. The summary line on stderr reports the files seen this run broken down by disposition, plus skips: @@ -726,9 +679,7 @@ output whether or not the scanned filesystem is still mounted. Processing: -- Records without a `content` hash (see "Database" above) are excluded: their - content is unknown, so they are never reported as duplicates. -- Group the remaining records by the key `(size, head, tail, content)`. +- Group the records by the key `(size, head, tail, content)`. - Every group with two or more paths is a duplicate group. - Within each group, sort paths lexicographically (byte order). The first path is the group's `first`; every other path is a `dupe`. @@ -782,9 +733,7 @@ records, split on `/`. Definitions: - A file's **signature** is `(size, head, tail, content)` — mtime is - informational and excluded. A record without a `content` hash has unknown - content: its signature is treated as unique to that file, so a tree containing - such a file never compares equal to any other tree. + informational and excluded. - A directory's **digest** is a SHA-256 Merkle digest computed bottom-up: serialize the directory's child entries — for a file child, its name and signature; for a subdirectory child, its name and that subdirectory's digest — @@ -858,8 +807,6 @@ database file size 2147483648 (2.0 GiB) modified 2026-10-16T12:02:50Z schema version 1 records 12345678 -records with head and tail hashes 1234567 -records with content hash 123456 size of recorded files 150000000000000 (136.4 TiB) ``` @@ -872,10 +819,8 @@ size of recorded files 150000000000000 (136.4 TiB) database is in WAL mode, the newest changes may be in the `-wal` file alone. - `schema version` is the database's `PRAGMA user_version`. A database with another version than this build's is refused, as by `report`. -- `records` is the number of records. `records with head and tail hashes` and - `records with content hash` count the records that have those hashes (see - "Database"), and `size of recorded files` adds up the sizes of all records, in - bytes, then in human units. +- `records` is the number of records. `size of recorded files` adds up the sizes + of all records, in bytes, then in human units. `info` runs in two phases, each with its own progress display on stderr (see "Progress"): `open` opens the database; `count` reads the schema version and @@ -891,28 +836,21 @@ Use the progress-bar library for all progress of `scan`, `report`, `trees` and stderr. `scan`, `report`, `trees` and `info` must always show that they are working and -not hung. Each phase gets its own display from the moment it starts, naming the -phase and its place among the command's phases and showing its progress, its -elapsed time and its estimated time remaining (ETA). +not hung. Each phase gets its own display from the moment it starts. The display +names the phase and its place among the command's phases. It shows the phase's +progress, its elapsed time and its time remaining. -A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`, -`[4/5] update`, `[5/5] content`): +A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`, +`[4/4] update`): - `load` counts the records first, then reads them against that count. While the count runs, the display says it is counting the records. - `walk` has no exact total. Its estimate is the number of records the last scan - left under the operands, known once `load` is done and shown with a `~`. Once - the walk passes the estimate, or when there are no such records, as on a first - scan, the ETA is unknown. + left under the operands, known once `load` is done and shown with a `~`. The + time remaining is unknown once the walk passes the estimate, and when there + are no such records, as on a first scan. - `hash` and `update` have exact totals. Only files that actually need hashing - appear in the hash total, so its ETA is meaningful. -- `content` first counts the records whose files it checks with `lstat`, which - SQLite does by adding up the number of records with each matching size, - `head`, and `tail`, without fetching the records. It then says it is searching - the database until SQLite returns the first record; the checks, and their rate - and ETA, start there. The files to read then get a display of their own, with - their count as its total, and so does the writing of the last of their records - to the database, with the count of those records as its total. + appear in the hash total, so its time remaining is meaningful. `trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`, `[4/4] write`, see "`trees` mode"): @@ -937,8 +875,8 @@ A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`, - `open` has no total. While it opens the database, waiting on a busy one included, the display says it is opening the database. - `count` has no total either, because one query counts everything: the display - says it is counting the records until the query returns, and its ETA is - unknown. + says it is counting the records until the query returns, and its time + remaining is unknown. Each display is one line holding: @@ -951,56 +889,69 @@ Each display is one line holding: while nothing completes, and a burst of items after a wait shows as a burst rather than as a steady rate. Once no item has completed for 10 seconds, the word `stalled` and where the phase waits take the rate's place (see below). -- elapsed time -- estimated time remaining, or `unknown`. It comes from the phase's average rate - since it started counting, so that it does not swing with each burst. +- elapsed time, labelled `elapsed` +- time remaining, labelled `remain`: a duration such as `12m9s`, or `unknown`. + It comes from the phase's average rate since it started counting, so that it + does not swing with each burst. -Example shape (exact layout is flexible, content is not): +Example lines, as printed when stderr is not a terminal: ``` -[3/5] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54 -[2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown -[3/5] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s +[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s +[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown +[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s ``` When no item has completed for 10 seconds, as while one large file is read, a disk is slow or the database is busy, the line says `stalled` and where the phase waits in place of the rate, which would read 0: `disk` for files, `db` for the database, `out` for stdout, and two of them, as `disk/db`, for a phase that -may wait on either. `walk` says `disk`. `hash` and the content phase's reads say -`disk/db`, as they read files and write their records to the database between -them, and so do the content phase's checks, which `lstat` each record's file as -the database search returns it. `load`, `update`, the content phase's last write -and the `read` phase of `trees` say `db`. `trees`' `write` says `out`, and its -`group`, which works in memory, says `cpu`. The `read` phase of `report` writes -the report while it reads the records, so it says `db/out`. A phase still -waiting for its total already says what it waits for, and says nothing more. +may wait on either. `walk` says `disk`. `hash` says `disk/db`, as it reads files +and writes their records to the database between them. `load`, `update` and the +`read` phase of `trees` say `db`. `trees`' `write` says `out`, and its `group`, +which works in memory, says `cpu`. The `read` phase of `report` writes the +report while it reads the records, so it says `db/out`. A phase still waiting +for its total already says what it waits for, and says nothing more. Additional requirements: -- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`, - the `open` phase of `report` and `trees`, and both phases of `info`, once it - has counted what it will do) is drawn as its line followed by the progress-bar - library's bar, 10 columns wide, which shows the percent in place of the line's - own. Any other phase, and a phase whose bar has no room, is drawn as the - library's spinner followed by its line. Each display is drawn again several - times a second, so its elapsed time keeps moving while a phase waits for its - next item. +- On a terminal, every display starts with the progress-bar library's spinner in + its first column, followed by its line. Each display is drawn again several + times a second, so its spinner turns and its elapsed time keeps moving while a + phase waits for its next item. +- On a terminal, a phase with an exact total ends its line with its percent and + a bar 10 columns wide. This percent takes the place of the line's own. The bar + stays at the terminal's right edge, so it does not move as the line grows or + shrinks. A phase with nothing to do shows no bar. Every phase has an exact + total once it has counted what it will do, except these: + - `scan`'s `walk`; + - the `open` phase of `report` and `trees`; + - both phases of `info`. + + On a terminal 100 columns wide: + + ``` + ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ | + ``` + - On a terminal no line reaches the terminal's last column, however narrow the - terminal is. A line too wide for the terminal first loses its bar, which gives - way to the spinner, then its rate, the words of a stalled phase, or what the - phase is waiting for. The phase and its place, the progress, the elapsed time - and the ETA are kept whole. On an 80-column terminal, at counts like those of - the stalled line in the example above, a stalled line keeps its words whenever - the elapsed time and the ETA are both in minutes, except in the content - phase's checks and reads. Their lines are the widest, and at most times in - minutes they lose the words. Where the words and the ETA do not both fit, as - at millions of files and times in hours, the words give way. Only when those - four alone are too wide, as on a terminal narrower than about 70 columns hours - into a scan of millions of files, is the line cut short at the terminal's - edge, so the end of the ETA is lost first. A terminal made narrower while a - phase runs gets the same: the next draw erases the line within the new width - and fits the line to it, and the display goes on redrawing in place. + terminal is. A line too wide for the terminal first loses whichever of these + it shows: its rate, the words of a stalled phase, or what the phase is waiting + for. The other four parts are kept whole: + - the phase and its place; + - the progress; + - the elapsed time; + - the time remaining. + + The line loses its bar only when those four and the bar do not fit. It is + cut short at the terminal's edge only when those four alone are too wide, + and then the end of the time remaining is lost first. That happens hours + into a scan of millions of files, on a terminal narrower than about 75 + columns. On a terminal 100 columns wide, the stalled line of the example + above keeps its words and its bar. The next draw fits the line to a terminal + made narrower while a phase runs. It erases the old line within the new + width, and the display goes on redrawing in place. + - When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI redraws: print the line the moment each phase starts, every 5 seconds while it runs (whether or not an item has completed, and never more often), and once @@ -1232,8 +1183,8 @@ Tracked in [TODO.md](TODO.md). ## Non-goals - No byte-for-byte compare, and no deletion or linking of duplicates. Files that - match are compared by a SHA-256 of the whole file below 50 MiB, and only by - samples at 50 MiB and over. The reports are advisory; acting on them is the + match are compared by a SHA-256 of the whole file below 100 MiB, and only by + samples at 100 MiB and over. The reports are advisory; acting on them is the user's job. - No persistence beyond the SQLite database described above; no export/import formats. diff --git a/TODO.md b/TODO.md index 4dd4dff..def2438 100644 --- a/TODO.md +++ b/TODO.md @@ -29,6 +29,36 @@ # Completed Steps +- an operand at or inside a directory named `.zfs` is never scanned, whether + given alone or under another operand. `scan` warns about it, counts it as + skipped and does not walk it (2026-10-26, + https://git.eeqj.de/sneak/sfdupes/issues/142) + +- a file or directory removed while the scan runs is not an error, in the walk + or in the hash phase. The scan prints no line for it and does not count it as + skipped. When the first of several hard links to a file is gone, the hash + phase reads the next one instead. On a terminal every progress display starts + with the spinner, the hash phase's included. A phase with an exact total ends + its line with its percent and a 10-column bar at the terminal's right edge, + which no longer moves as the line's length changes. A line too wide for the + terminal now loses its rate before its bar. The time remaining is labelled + `remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133) + +- the `head`, `tail` and `content` columns are BLOBs, each holding the raw 32 + bytes of a SHA-256, half the size of the hex text they held. The hashes stay + raw from the hash phase through the database to `report` and `trees`. Reading + a record whose hash is not 32 bytes is an error (2026-10-26, + https://git.eeqj.de/sneak/sfdupes/issues/134) + +- the hash phase gives every file the walk records three hashes: `head` and + `tail`, the SHA-256 of its first and last 64 KiB, and `content`, the SHA-256 + of the whole file under 100 MiB and of 1 MiB at each gigabyte from 100 MiB up. + The 10 MiB and 50 MiB thresholds, the size census, the records stored without + hashes and the content phase are gone, so a scan has four phases and a rescan + reads only new and changed files. `info` no longer counts the records with + hashes, since every record has them (2026-10-25, + https://git.eeqj.de/sneak/sfdupes/issues/132) + - re-vendor `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` from `sneak/prompts` `next` at `0653579`: in the `lint`, `test` and `builder` stages of the `Dockerfile` the module download, and in `builder` the diff --git a/cancel_test.go b/cancel_test.go index 20e72e2..577c54d 100644 --- a/cancel_test.go +++ b/cancel_test.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -168,8 +169,8 @@ func assertRecordsIntact(t *testing.T, db *sql.DB, before []string) { // TestSyncScanCancelledMidWalkKeepsRecords is the regression net under // the post-walk guard. The scan is cancelled part-way through the -// walk, so it reaches the guard holding a genuinely partial size -// census and a still-populated index of records the walk never got to. +// walk, so it reaches the guard having walked a genuinely partial tree +// and holding a still-populated index of records the walk never got to. // Every one of those records would look vanished to the update phase. // The guard is what stops the scan there, and this test is what // notices if it stops doing so: deleting the guard, or making it @@ -208,10 +209,10 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) { } // assertWalkGuardAborted checks that the scan stopped at the post-walk -// guard: with a census that is neither empty (the walk really ran) -// nor complete (it really was cut short), and with no record counted -// for removal. A removal count means the partial census was carried -// past the guard into the update phase, which is the failure this test +// guard: with a walk that reached some files (it really ran) but not +// all of them (it really was cut short), and with no record counted for +// removal. A removal count means the partial walk was carried past +// the guard into the update phase, which is the failure this test // exists to catch. func assertWalkGuardAborted(t *testing.T, st scanStats, err error) { t.Helper() @@ -222,30 +223,30 @@ func assertWalkGuardAborted(t *testing.T, st scanStats, err error) { } if st.unchanged == 0 { - t.Fatalf("stats = %+v: the census is empty, so the walk never "+ + t.Fatalf("stats = %+v: the walk reached no file, so it never "+ "ran and the guard was reached for the wrong reason", st) } if st.unchanged >= walkCancelFiles { - t.Fatalf("stats = %+v: the census covers the whole tree, so the "+ - "walk was not cut short", st) + t.Fatalf("stats = %+v: the walk reached the whole tree, so it "+ + "was not cut short", st) } // The workers drop every directory still queued once the scan is // cancelled, so only the files in the directories already in flight - // can add to the census after the fact. A census beyond that bound + // can still be reached after the fact. A walk beyond that bound // would mean the cancellation was not observed where it should have // been. limit := walkCancelAtDone + walkCancelInFlightFiles if st.unchanged > limit { - t.Errorf("census covers %d files, want at most %d: the walk kept "+ + t.Errorf("the walk reached %d files, want at most %d: it kept "+ "taking directories off the queue after cancellation", st.unchanged, limit) } if st.removed != 0 { t.Errorf("stats = %+v: the scan counted records for removal from "+ - "a partial census", st) + "a partial walk", st) } } @@ -762,7 +763,9 @@ func TestHashWorkerDropsQueuedRuns(t *testing.T) { var hashed atomic.Bool - hash := func(path string, size int64) (string, string, string, error) { + hash := func(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error, + ) { hashed.Store(true) return hashSignature(path, size) @@ -802,7 +805,9 @@ func TestHashWorkerAbandonsBlockedSend(t *testing.T) { // The scan is cancelled while the worker hashes, so the worker has // already passed the check that drops queued runs. - hash := func(path string, size int64) (string, string, string, error) { + hash := func(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error, + ) { cancel() return hashSignature(path, size) diff --git a/db.go b/db.go index de61db5..c7835b4 100644 --- a/db.go +++ b/db.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -48,16 +49,17 @@ const lockFilePerm = 0o600 // createTableSQL is the schema applied to a fresh database. Paths are // BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime // holds whole Unix seconds and mtime_nsec the nanoseconds within that -// second. +// second. head, tail and content each hold the raw 32 bytes of a +// SHA-256. const createTableSQL = ` CREATE TABLE files ( path BLOB PRIMARY KEY, size INTEGER NOT NULL, mtime INTEGER NOT NULL, mtime_nsec INTEGER NOT NULL, - head TEXT NOT NULL, - tail TEXT NOT NULL, - content TEXT NOT NULL + head BLOB NOT NULL, + tail BLOB NOT NULL, + content BLOB NOT NULL ) WITHOUT ROWID ` @@ -92,6 +94,10 @@ var errSchemaVersion = errors.New("unsupported database schema version") // database. var errScanRunning = errors.New("another scan is running") +// errHashSize reports a record whose head, tail or content is not the +// 32 bytes of a SHA-256. +var errHashSize = errors.New("hash is not 32 bytes") + // databasePath resolves the database location: SFDUPES_DATABASE when // set and non-empty, the compiled-in default otherwise. It returns the // path, with the working directory, its symlinks resolved, put in front @@ -492,19 +498,24 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64), for rows.Next() { var ( - path []byte - sec, nsec int64 - r scanRec + path, head, tail, content []byte + sec, nsec int64 + r scanRec ) - err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail, - &r.content) + err = rows.Scan(&path, &r.size, &sec, &nsec, &head, &tail, &content) if err != nil { return fmt.Errorf("read record: %w", err) } r.path = string(path) r.mtime = time.Unix(sec, nsec) + + err = r.setHashes(head, tail, content) + if err != nil { + return fmt.Errorf("read record %s: %w", escapePath(r.path), err) + } + fn(r) } @@ -561,17 +572,22 @@ func loadSignatureRows(ctx context.Context, db *sql.DB, for rows.Next() { var ( - path []byte - r scanRec + path, head, tail, content []byte + r scanRec ) - err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path) + err = rows.Scan(&r.size, &head, &tail, &content, &path) if err != nil { return fmt.Errorf("read record: %w", err) } r.path = string(path) + err = r.setHashes(head, tail, content) + if err != nil { + return fmt.Errorf("read record %s: %w", escapePath(r.path), err) + } + err = fn(r) if err != nil { return err @@ -586,6 +602,22 @@ func loadSignatureRows(ctx context.Context, db *sql.DB, return nil } +// setHashes sets r's head, tail and content to the values read from +// those columns. Each must be the raw 32 bytes of a SHA-256. +func (r *scanRec) setHashes(head, tail, content []byte) error { + for _, b := range [][]byte{head, tail, content} { + if len(b) != sha256.Size { + return fmt.Errorf("%w: found %d bytes", errHashSize, len(b)) + } + } + + r.head = [sha256.Size]byte(head) + r.tail = [sha256.Size]byte(tail) + r.content = [sha256.Size]byte(content) + + return nil +} + // countRecords returns the number of records in the database. func countRecords(ctx context.Context, db *sql.DB) (int64, error) { var n int64 @@ -598,15 +630,11 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) { return n, nil } -// databaseCountsSQL counts the records, those with head and tail -// hashes and those with a content hash, and adds up their sizes, in one +// databaseCountsSQL counts the records and adds up their sizes, in one // pass over the table. A SUM over no rows is NULL, so on an empty table -// each SUM falls back to 0. +// the SUM falls back to 0. const databaseCountsSQL = ` -SELECT COUNT(*), - COALESCE(SUM(head <> '' AND tail <> ''), 0), - COALESCE(SUM(content <> ''), 0), - COALESCE(SUM(size), 0) +SELECT COUNT(*), COALESCE(SUM(size), 0) FROM files ` @@ -615,8 +643,6 @@ FROM files type databaseCounts struct { schemaVersion int records int64 // every record - headTail int64 // records with head and tail hashes - content int64 // records with a content hash size int64 // the sizes of all records added up } @@ -644,7 +670,7 @@ func readDatabaseCounts(ctx context.Context, } err = tx.QueryRowContext(ctx, databaseCountsSQL). - Scan(&c.records, &c.headTail, &c.content, &c.size) + Scan(&c.records, &c.size) if err != nil { return c, fmt.Errorf("count records: %w", err) } @@ -652,15 +678,15 @@ func readDatabaseCounts(ctx context.Context, return c, nil } -// loadFileMeta streams every record's path, size, mtime, and whether -// it carries hashes to fn. Scan change detection needs no hash -// values, and skipping the hash columns keeps the scan's in-memory -// index small on multi-million-file databases. +// loadFileMeta streams every record's path, size and mtime to fn. Scan +// change detection needs no hash values, and skipping the hash columns +// keeps the scan's in-memory index small on multi-million-file +// databases. func loadFileMeta(ctx context.Context, db *sql.DB, - fn func(path string, size int64, mtime time.Time, hashed bool), + fn func(path string, size int64, mtime time.Time), ) error { rows, err := db.QueryContext(ctx, - "SELECT path, size, mtime, mtime_nsec, head <> '' FROM files") + "SELECT path, size, mtime, mtime_nsec FROM files") if err != nil { return fmt.Errorf("read records: %w", err) } @@ -671,94 +697,14 @@ func loadFileMeta(ctx context.Context, db *sql.DB, var ( path []byte size, sec, nsec int64 - hashed int64 ) - err = rows.Scan(&path, &size, &sec, &nsec, &hashed) + err = rows.Scan(&path, &size, &sec, &nsec) if err != nil { return fmt.Errorf("read record: %w", err) } - fn(string(path), size, time.Unix(sec, nsec), hashed != 0) - } - - err = rows.Err() - if err != nil { - return fmt.Errorf("read records: %w", err) - } - - return nil -} - -// contentGroupsSQL selects each group (the records sharing a size, -// head, and tail) of records of at least headTailMin bytes that has more -// than one record and at least one record without a content hash, with -// the number of its records. -const contentGroupsSQL = ` -SELECT size, head, tail, COUNT(*) AS records -FROM files -WHERE size >= ? AND head <> '' -GROUP BY size, head, tail -HAVING COUNT(*) > 1 AND SUM(content = '') > 0 -` - -// contentCandidatesSQL selects every record of the groups -// contentGroupsSQL selects, with whether each record has a content -// hash. SQLite does the grouping, so no other record's hashes are -// loaded into memory; the rows come ordered by size, head, and tail, so -// each group's rows arrive together. -const contentCandidatesSQL = ` -SELECT f.path, f.size, f.mtime, f.mtime_nsec, f.head, f.tail, f.content <> '' -FROM files AS f -JOIN (` + contentGroupsSQL + `) AS g USING (size, head, tail) -ORDER BY size, head, tail -` - -// countContentCandidates returns the number of rows of -// contentCandidatesSQL, adding up the records of the groups -// contentGroupsSQL selects, so it neither joins nor sorts. -func countContentCandidates(ctx context.Context, db *sql.DB) (int64, error) { - var n int64 - - err := db.QueryRowContext(ctx, - "SELECT COALESCE(SUM(records), 0) FROM ("+contentGroupsSQL+")", - headTailMin).Scan(&n) - if err != nil { - return 0, fmt.Errorf("count content candidates: %w", err) - } - - return n, nil -} - -// loadContentCandidates streams the rows of contentCandidatesSQL to fn: -// each record, without its content hash, and whether it has one. -func loadContentCandidates(ctx context.Context, db *sql.DB, - fn func(r scanRec, hashed bool), -) error { - rows, err := db.QueryContext(ctx, contentCandidatesSQL, headTailMin) - if err != nil { - return fmt.Errorf("read records: %w", err) - } - - defer func() { _ = rows.Close() }() - - for rows.Next() { - var ( - path []byte - sec, nsec int64 - r scanRec - hashed int64 - ) - - err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail, - &hashed) - if err != nil { - return fmt.Errorf("read record: %w", err) - } - - r.path = string(path) - r.mtime = time.Unix(sec, nsec) - fn(r, hashed != 0) + fn(string(path), size, time.Unix(sec, nsec)) } err = rows.Err() @@ -842,7 +788,8 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec, for _, r := range upserts { _, err = st.ExecContext(ctx, []byte(r.path), r.size, - r.mtime.Unix(), r.mtime.Nanosecond(), r.head, r.tail, r.content) + r.mtime.Unix(), r.mtime.Nanosecond(), + r.head[:], r.tail[:], r.content[:]) if err != nil { return fmt.Errorf("upsert %s: %w", r.path, err) } diff --git a/db_test.go b/db_test.go index 427fb61..9268048 100644 --- a/db_test.go +++ b/db_test.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -473,6 +474,67 @@ func TestLoadRowsCountAgreeDuringScan(t *testing.T) { } } +// TestLoadRowsRefuseWrongSizeHash stores a record whose head, tail or +// content is not the 32 bytes of a SHA-256: either one byte, or the 64 +// bytes of a SHA-256 written as hex text. loadFileRows and +// loadSignatureRows must each fail on it with an error naming its path. +func TestLoadRowsRefuseWrongSizeHash(t *testing.T) { + t.Parallel() + + hexText := strings.Repeat("ab", sha256.Size) + + cases := []struct { + name string + update string + value any + }{ + {"one-byte head", "UPDATE files SET head = ?", []byte{0}}, + {"hex-text head", "UPDATE files SET head = ?", hexText}, + {"one-byte tail", "UPDATE files SET tail = ?", []byte{0}}, + {"hex-text tail", "UPDATE files SET tail = ?", hexText}, + {"one-byte content", "UPDATE files SET content = ?", []byte{0}}, + {"hex-text content", "UPDATE files SET content = ?", hexText}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + t.Parallel() + + db := openTestDB(t) + + err := applyChanges(t.Context(), db, + []scanRec{{path: "/a/f"}}, nil, nil) + if err != nil { + t.Fatal(err) + } + + _, err = db.ExecContext(t.Context(), c.update, c.value) + if err != nil { + t.Fatal(err) + } + + loads := map[string]func() error{ + "loadFileRows": func() error { + return loadFileRows(t.Context(), db, func(int64) {}, + func(scanRec) {}) + }, + "loadSignatureRows": func() error { + return loadSignatureRows(t.Context(), db, func(int64) {}, + func(scanRec) error { return nil }) + }, + } + + for name, load := range loads { + err := load() + if !errors.Is(err, errHashSize) || + !strings.Contains(err.Error(), "/a/f") { + t.Errorf("%s: err = %v, want %v naming /a/f", + name, err, errHashSize) + } + } + }) + } +} + // TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL // from the files_signature index alone, with nothing to sort: the // first record comes at once, so report's read phase counts each @@ -527,12 +589,13 @@ func TestApplyChangesRoundTrip(t *testing.T) { // written. recs := []scanRec{ { - size: 2, mtime: time.Unix(20, 999_999_999), head: "h2", tail: "t2", - content: "c2", path: "/a/tab\tnew\nline", + size: 2, mtime: time.Unix(20, 999_999_999), head: testHash("h2"), + tail: testHash("t2"), content: testHash("c2"), + path: "/a/tab\tnew\nline", }, { - size: 1, mtime: time.Unix(10, 0), head: "h1", tail: "t1", - content: "c1", path: "/a/x", + size: 1, mtime: time.Unix(10, 0), head: testHash("h1"), + tail: testHash("t1"), content: testHash("c1"), path: "/a/x", }, } @@ -553,8 +616,8 @@ func TestApplyChangesRoundTrip(t *testing.T) { // An upsert for an existing path updates in place; a delete // removes exactly its path. upd := scanRec{ - size: 3, mtime: time.Unix(30, 0), head: "h3", tail: "t3", content: "c3", - path: "/a/x", + size: 3, mtime: time.Unix(30, 0), head: testHash("h3"), + tail: testHash("t3"), content: testHash("c3"), path: "/a/x", } err = applyChanges(t.Context(), db, []scanRec{upd}, @@ -581,8 +644,8 @@ func TestApplyChangesBatching(t *testing.T) { recs := make([]scanRec, 0, n) for i := range n { recs = append(recs, scanRec{ - size: int64(i), mtime: time.Unix(1, 0), head: "h", tail: "t", - path: fmt.Sprintf("/batch/%07d", i), + size: int64(i), mtime: time.Unix(1, 0), head: testHash("h"), + tail: testHash("t"), path: fmt.Sprintf("/batch/%07d", i), }) } @@ -612,62 +675,3 @@ func TestApplyChangesBatching(t *testing.T) { t.Fatalf("records = %d, want 0", len(got)) } } - -// TestCountContentCandidates checks that countContentCandidates counts -// the records the content phase's search returns: those of each group -// of two or more records of headTailMin or more with a head, where at -// least one has no content hash. -func TestCountContentCandidates(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - - // Each group's records: size, head, and content, one per record. - groups := []struct { - size int64 - head string - contents []string - }{ - {headTailMin, "h", []string{"", "c", "c"}}, // returned - {headTailMin + 1, "h", []string{"", ""}}, // returned - {headTailMin + 2, "h", []string{"c", "c"}}, // all hashed - {headTailMin + 3, "h", []string{""}}, // one record - {headTailMin - 1, "h", []string{"", ""}}, // too small - {headTailMin + 4, "", []string{"", ""}}, // no head - } - - var recs []scanRec - - for _, g := range groups { - for _, content := range g.contents { - recs = append(recs, scanRec{ - size: g.size, mtime: time.Unix(1, 0), head: g.head, - tail: "t", content: content, - path: fmt.Sprintf("/r/%d", len(recs)), - }) - } - } - - err := applyChanges(t.Context(), db, recs, nil, nil) - if err != nil { - t.Fatalf("applyChanges: %v", err) - } - - got, err := countContentCandidates(t.Context(), db) - if err != nil { - t.Fatalf("countContentCandidates: %v", err) - } - - var returned int64 - - err = loadContentCandidates(t.Context(), db, - func(scanRec, bool) { returned++ }) - if err != nil { - t.Fatalf("loadContentCandidates: %v", err) - } - - if got != 5 || returned != 5 { - t.Errorf("count = %d, records returned = %d, want 5 and 5", - got, returned) - } -} diff --git a/exclude_test.go b/exclude_test.go index 20f195d..b6b48c7 100644 --- a/exclude_test.go +++ b/exclude_test.go @@ -164,9 +164,8 @@ func TestDefaultExcludesHelp(t *testing.T) { // TestScanDoesNotEnterDefaultExclude checks that the walk does not // enter a default exclude below an operand. It says so on a line of its // own and does not count it as skipped. The records below it count as -// outside the operands: the record of a file gone from it is kept, and -// its size joins the size census, while a file added to it is not -// recorded. +// outside the operands: the record of a file gone from it is kept, +// while a file added to it is not recorded. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestScanDoesNotEnterDefaultExclude(t *testing.T) { @@ -186,10 +185,6 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) { unseen := writeFile(t, excluded, "new", pattern(2, 10)) - // The same size as gone, so it is hashed only if the census holds the - // size of gone's record. - added := writeFile(t, dir, "added", pattern(3, 100)) - st, err := syncScan(t.Context(), db, []string{dir}, 1, false, walkRules{excludes: []string{excluded}}) if err != nil { @@ -203,11 +198,6 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) { recs := dbRecords(t, db) recordByPath(t, recs, gone) - if recordByPath(t, recs, added).head == "" { - t.Errorf("%s not hashed: the census lacks the size of %s", - added, gone) - } - if slices.Contains(recordPaths(recs), unseen) { t.Errorf("%s recorded: the walk entered %s", unseen, excluded) } @@ -391,9 +381,8 @@ func TestRunScanSkipsNonPersistentOperand(t *testing.T) { // TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem // that is not persistent beside one on the tests' temporary directory. // The records an earlier scan left in it count as outside the operands: -// the record of a file gone from it is kept, and its size joins the size -// census, while a file added to it is not recorded, and its size does -// not join the census. +// the record of a file gone from it is kept, while a file added to it +// is not recorded. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestScanKeepsRecordsOnNonPersistent(t *testing.T) { @@ -413,11 +402,6 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) { unseen := writeFile(t, shm, "new", pattern(2, 10)) - // added has the size of gone, and lonely that of unseen: each is - // hashed only if the census holds that size. - added := writeFile(t, dir, "added", pattern(3, 100)) - lonely := writeFile(t, dir, "lonely", pattern(4, 10)) - st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules) if err != nil { t.Fatal(err) @@ -430,15 +414,6 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) { recs := dbRecords(t, db) recordByPath(t, recs, gone) - if recordByPath(t, recs, added).head == "" { - t.Errorf("%s not hashed: the census lacks the size of %s", - added, gone) - } - - if recordByPath(t, recs, lonely).head != "" { - t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen) - } - if slices.Contains(recordPaths(recs), unseen) { t.Errorf("%s recorded", unseen) } @@ -450,51 +425,6 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) { } } -// TestScanContentSkipsNonPersistent checks that the content phase does -// not read a file on a filesystem that is not persistent, though an -// earlier scan left a record for it: it is treated as a changed file, -// so its record is left as it is and its identical copy in another -// tree, having no other match, is not read either. -func TestScanContentSkipsNonPersistent(t *testing.T) { - t.Parallel() - - shm, _, rules := shmTempDir(t) - db := openTestDB(t) - - // A scan that, without the list of filesystems, records a with a - // head and tail but no content hash. - a := sparseFileWithoutMatch(t, shm, "a", headTailMin) - - syncTree(t, db, shm) - - before := recordByPath(t, dbRecords(t, db), a) - if before.head == "" || before.content != "" { - t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before) - } - - b := sparseFile(t, t.TempDir(), "b", headTailMin) - - st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false, - rules) - if err != nil { - t.Fatal(err) - } - - if st.skipped != 0 { - t.Errorf("skipped = %d, want 0", st.skipped) - } - - recs := dbRecords(t, db) - if r := recordByPath(t, recs, a); r != before { - t.Errorf("record = %+v, want it left as %+v", r, before) - } - - if r := recordByPath(t, recs, b); r.content != "" { - t.Errorf("%s: content = %q, want none: its only match is on %s", - b, r.content, shm) - } -} - // TestWalkLeavesOutNonPersistent starts the walk itself in a directory // on a filesystem that is not persistent, which a scan never does: it // drops such an operand first. Below a directory on a persistent diff --git a/info.go b/info.go index 4f705c4..3ec91ff 100644 --- a/info.go +++ b/info.go @@ -113,9 +113,6 @@ func writeInfo(stdout io.Writer, dbPath, source string, dbFile.ModTime().UTC().Format(time.RFC3339)) fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion) fmt.Fprintf(&b, "records\t%d\n", counts.records) - fmt.Fprintf(&b, "records with head and tail hashes\t%d\n", - counts.headTail) - fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content) fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n", counts.size, humanBytes(counts.size)) diff --git a/info_test.go b/info_test.go index 1d308ed..5d6a58e 100644 --- a/info_test.go +++ b/info_test.go @@ -12,23 +12,25 @@ import ( // infoOpenLine is the plain line of info's open phase when it opens the // database at once; it prints it as the phase starts and as it ends. const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" // infoCountLine is the plain line of info's count phase when the count // takes under a second; it prints it as the phase starts and as it // ends. const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " + - "eta unknown" + "remain unknown" -// infoRecs are four records: a duplicate pair with every hash, a file -// of 20 MiB with head and tail hashes but no content hash yet, and a -// file never hashed. +// infoRecs are four records: a duplicate pair, a file of 20 MiB and +// another small file. func infoRecs() []scanRec { + h, tl, c, u := testHash("h"), testHash("t"), testHash("c"), testHash("u") + bigH, bigT, bigC := testHash("bh"), testHash("bt"), testHash("bc") + return []scanRec{ - {size: 5, head: "h", tail: "t", content: "c", path: "/d/a"}, - {size: 5, head: "h", tail: "t", content: "c", path: "/d/b"}, - {size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"}, - {size: 7, path: "/d/unique"}, + {size: 5, head: h, tail: tl, content: c, path: "/d/a"}, + {size: 5, head: h, tail: tl, content: c, path: "/d/b"}, + {size: 20 << 20, head: bigH, tail: bigT, content: bigC, path: "/d/big"}, + {size: 7, head: u, tail: u, content: u, path: "/d/unique"}, } } @@ -45,16 +47,12 @@ func TestRunInfo(t *testing.T) { "empty": { want: "schema version\t1\n" + "records\t0\n" + - "records with head and tail hashes\t0\n" + - "records with content hash\t0\n" + "size of recorded files\t0 (0 B)\n", }, "filled": { recs: infoRecs(), want: "schema version\t1\n" + "records\t4\n" + - "records with head and tail hashes\t3\n" + - "records with content hash\t2\n" + "size of recorded files\t20971537 (20.0 MiB)\n", }, } diff --git a/main.go b/main.go index 2178408..7ccb09f 100644 --- a/main.go +++ b/main.go @@ -1,14 +1,12 @@ // Command sfdupes quickly identifies candidate duplicate files across // very large filesystems without reading every byte of every file. // Files are considered duplicates when their sizes are equal and they -// agree on a short ladder of SHA-256 hashes. A file under 10 MiB is -// hashed in full. A larger file is compared on the hashes of its first -// and last 64 KiB, and only when those match another file's is its -// content hash computed and compared: of the whole file when it is -// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB -// or larger. scan maintains a persistent SQLite database of file -// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) -// that the other subcommands read. +// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64 +// KiB, and of the content. The content hash covers the whole file under +// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large +// file is never read in full. scan maintains a persistent SQLite +// database of file signatures (SFDUPES_DATABASE, default +// /var/lib/sfdupes/db.sqlite) that the other subcommands read. // // Usage: // @@ -238,7 +236,7 @@ func newScanCommand() *cobra.Command { }), } scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(), - "concurrent workers for the walk, hash, and content phases") + "concurrent workers for the walk and hash phases") scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false, "do not cross filesystem boundaries") scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes", diff --git a/main_test.go b/main_test.go index 8e39eab..cca359a 100644 --- a/main_test.go +++ b/main_test.go @@ -928,24 +928,21 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) { } func TestRunScanSkipsZFSOperand(t *testing.T) { - path := testDBPath(t) - t.Setenv(databaseEnv, path) - zfs := filepath.Join(t.TempDir(), ".zfs") - snapshot := filepath.Join(zfs, "snapshot", "hourly") - f := writeFile(t, snapshot, "f", pattern(1, 10)) + f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10)) - // An operand beneath a .zfs directory is walked, because it is not - // itself named .zfs. - scanOK(t, snapshot) + // No scan records a file inside a .zfs directory, so the record + // beneath the operand is written straight into the database. + path := seedDatabase(t, []scanRec{{size: 10, path: f}}) + t.Setenv(databaseEnv, path) assertOperandSkipped(t, path, zfs, ".zfs directory", f) } // assertOperandSkipped scans operand alone and checks that it is skipped // as kind: a warning naming it, one skip in the summary, exit 0, and the -// record for kept, which an earlier scan stored beneath operand, still -// in the database at dbPath. +// record for kept, stored beneath operand before this scan, still in the +// database at dbPath. func assertOperandSkipped(t *testing.T, dbPath, operand, kind, kept string, ) { @@ -974,6 +971,121 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind, recordByPath(t, dbRecords(t, db), kept) } +// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs +// directory, first alone and then under another operand. Each time it +// is warned about and counted as skipped, and nothing in it is walked. +// The other operand is scanned as usual. +func TestRunScanSkipsOperandInsideZFS(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := t.TempDir() + hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly") + writeFile(t, hourly, "f", pattern(1, 10)) + f := writeFile(t, dir, "f", pattern(2, 10)) + + warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n" + + for _, tc := range []struct { + operands []string + summary string + }{ + { + []string{hourly}, + "scan: 0 files seen (0 added, 0 updated, 0 removed, " + + "0 unchanged), 1 skipped\n", + }, + { + []string{dir, hourly}, + "scan: 1 files seen (1 added, 0 updated, 0 removed, " + + "0 unchanged), 1 skipped\n", + }, + } { + stderr := scanOK(t, tc.operands...) + if !strings.Contains(stderr, warning) || + !strings.Contains(stderr, tc.summary) { + t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands, + stderr, warning, tc.summary) + } + } + + db, err := openDB(path, reportParams) + if err != nil { + t.Fatal(err) + } + + t.Cleanup(func() { _ = db.Close() }) + + if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) { + t.Errorf("records = %q, want %q", got, []string{f}) + } +} + +// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where +// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about +// by the path given, counted as skipped, and not walked. +func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) { + t.Setenv(databaseEnv, testDBPath(t)) + + dir := t.TempDir() + writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10)) + + snaps := filepath.Join(dir, "snaps") + + err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps) + if err != nil { + t.Fatal(err) + } + + daily := filepath.Join(snaps, "daily") + stderr := scanOK(t, daily) + + warning := "walk " + daily + ": skipping operand inside a .zfs directory\n" + summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " + + "0 unchanged), 1 skipped\n" + + if !strings.Contains(stderr, warning) || + !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary) + } +} + +// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is +// a symlink to a directory outside any .zfs directory. The operand is +// scanned as usual, and its file is recorded by the path given. +func TestRunScanWalksOperandThroughSymlink(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := t.TempDir() + writeFile(t, dir, "pool/data/daily/f", pattern(1, 10)) + + data := filepath.Join(dir, "data") + + err := os.Symlink(filepath.Join(dir, "pool", "data"), data) + if err != nil { + t.Fatal(err) + } + + daily := filepath.Join(data, "daily") + stderr := scanOK(t, daily) + + summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " + + "0 unchanged), 0 skipped\n" + if !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q", stderr, summary) + } + + db, err := openDB(path, reportParams) + if err != nil { + t.Fatal(err) + } + + t.Cleanup(func() { _ = db.Close() }) + + recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f")) +} + func TestRunReportSucceeds(t *testing.T) { path := testDBPath(t) t.Setenv(databaseEnv, path) diff --git a/progress.go b/progress.go index 6802825..85e79d4 100644 --- a/progress.go +++ b/progress.go @@ -9,6 +9,7 @@ import ( "strings" "sync" "time" + "unicode/utf8" "github.com/schollz/progressbar/v3" "golang.org/x/term" @@ -34,10 +35,17 @@ const stallTime = 10 * time.Second // spinnerType selects the progressbar library's spinner style. const spinnerType = 14 -// barWidth is the width of the progressbar library's bar between its -// two ends, as in the example line in README "Progress". +// barWidth is the width of a phase's bar between its two ends, as in +// the example line in README "Progress". const barWidth = 10 +// barCell is one column of the filled part of a bar. +const barCell = "█" + +// barColumns is the width of what percentBar returns: the percent in +// four columns and its sign, a space, and the bar with its two ends. +const barColumns = barWidth + 8 + // percentScale converts a fraction to a percentage. const percentScale = 100 @@ -45,17 +53,12 @@ const percentScale = 100 // read, as the progressbar library takes it. const fallbackWidth = 80 -// spinnerMargin is the columns of a spinner's line besides its text: -// the spinner and the three spaces the library draws around the text, -// and the terminal's last column, which every line stays clear of so -// that no terminal wraps it. +// spinnerMargin is the columns of a line besides its text: the spinner +// and the three spaces the library draws around the text, and the +// terminal's last column, which every line stays clear of so that no +// terminal wraps it. const spinnerMargin = 5 -// barMargin is the columns of a bar's line besides its text: the -// percent and the bar the library draws after the text, the spaces -// around them, and the terminal's last column. -const barMargin = barWidth + 11 - // countingRecords is what the display of a phase that reads the records // says while it counts them, before it has its total. const countingRecords = "counting the records" @@ -85,7 +88,7 @@ func terminalWidth() int { return width } -// phaseLabel names phase with its place among phases, as "[2/5] walk". +// phaseLabel names phase with its place among phases, as "[2/4] walk". func phaseLabel(phases []string, phase string) string { return fmt.Sprintf("[%d/%d] %s", slices.Index(phases, phase)+1, len(phases), phase) @@ -109,13 +112,13 @@ type drawnCount struct { } // progress renders one phase's progress on stderr. On a TTY the -// progressbar library draws line, fitted to the terminal's width: after -// the library's spinner, or, for a phase with an exact total, before the -// library's bar when the terminal has room for the bar beside it. It is -// drawn again every redrawInterval, so its elapsed time keeps moving -// while the phase waits for its next item. When stderr is not a TTY it -// emits no ANSI redraws: it prints line as the phase starts, every -// plainInterval while it runs, and as it ends. +// progressbar library draws its spinner and then line, fitted to the +// terminal's width. A phase with an exact total ends the line with its +// percent and bar, at the terminal's right edge, when the terminal has +// room for them. It is drawn again every redrawInterval, so its elapsed +// time keeps moving while the phase waits for its next item. When +// stderr is not a TTY it emits no ANSI redraws: it prints line as the +// phase starts, every plainInterval while it runs, and as it ends. // // The draws come from a goroutine of their own, so mu guards the fields // below it. Every method must be called from the main goroutine. A nil @@ -131,7 +134,6 @@ type progress struct { mu sync.Mutex bar *progressbar.ProgressBar // what the library draws; nil until then - withBar bool // bar is the library's bar, not its spinner widest int // the widest line bar has drawn, in columns label string unit string @@ -141,7 +143,7 @@ type progress struct { stalledOn string count int64 start time.Time // when the phase started, for the elapsed time - rateStart time.Time // when the count started, for the ETA + rateStart time.Time // when the count started, for the time remaining lastItem time.Time // when the last item was completed, or the count started // drawn holds the count at each draw over the last rateWindow, oldest // first; the rate counts from the first. It keeps the draw before the @@ -215,55 +217,34 @@ func (p *progress) draw() { } if p.width == nil { - fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true)) + fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, false)) return } width := p.width() + text := p.line(now, width-spinnerMargin, true) - // A phase with an exact total gets the library's bar when its whole - // line, without the percent the bar shows, fits beside the bar: the - // bar gives way before any part of the line. A total of 0 leaves the - // bar nothing to fill. - text, withBar := "", false - if p.total > 0 && !p.estimate { - text = p.line(now, math.MaxInt, false) - withBar = len(text) <= width-barMargin - } - - margin := barMargin - if !withBar { - margin = spinnerMargin - text = p.line(now, width-margin, true) - } - - // The library makes a bar or a spinner, and before each draw erases - // its line with as many spaces as the widest line it has drawn. A - // display gets a new one when it turns from one to the other, as its - // line grows or the terminal is resized, and when the terminal has - // been made too narrow for that erase, which would wrap and leave a - // line behind at every draw. - if p.bar == nil || p.withBar != withBar || p.widest >= width { - p.newBar(withBar, width) - } - - if withBar { - _ = p.bar.Set64(p.count) + // Before each draw the library erases its line with as many spaces as + // the widest line it has drawn. A display gets a new one when the + // terminal has been made too narrow for that erase, which would wrap + // and leave a line behind at every draw. + if p.bar == nil || p.widest >= width { + p.newBar(width) } p.bar.Describe(text) - // The library drew the text and its margin, less the terminal's last - // column, which the margin keeps clear. - p.widest = max(p.widest, len(text)+margin-1) + // The library drew the spinner, the text and the spaces around it, + // less the terminal's last column, which spinnerMargin keeps clear. A + // bar's cells take one column each but more than one byte. + p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1) } -// newBar has the library make what draw draws: a bar for the phase's -// total when withBar is set, and a spinner when not. Either shows the -// text draw gives it, and nothing of its own besides the spinner, or -// the bar and its percent. p.mu must be held. -func (p *progress) newBar(withBar bool, width int) { +// newBar has the library make what draw draws: its spinner, followed by +// the text draw gives it and nothing of the library's own. p.mu must be +// held. +func (p *progress) newBar(width int) { if p.bar != nil { // Erase what was drawn before, short of the last column of a // terminal that may have been made narrower since: the library's @@ -272,29 +253,36 @@ func (p *progress) newBar(withBar bool, width int) { fmt.Fprint(os.Stderr, "\r"+blank+"\r") } - total := int64(-1) // the library's spinner - if withBar { - total = p.total - } - - p.bar = progressbar.NewOptions64(total, + // A total of -1 gives the library's spinner. + p.bar = progressbar.NewOptions64(-1, progressbar.OptionSetWriter(os.Stderr), - progressbar.OptionSetWidth(barWidth), progressbar.OptionSpinnerType(spinnerType), // The spinner turns at each draw, not on a timer of the // library's own. progressbar.OptionSetSpinnerChangeInterval(0), - // The text carries the elapsed time and the ETA. + // The text carries the elapsed time and the time remaining. progressbar.OptionSetElapsedTime(false), progressbar.OptionSetPredictTime(false), ) - p.withBar, p.widest = withBar, 0 + p.widest = 0 +} + +// percentBar returns the percent and the bar that end the text of a +// phase with an exact total, as " 25% |██ |": the bar's +// barWidth columns fill in proportion to the items done. A count past +// the total, as when a scan adds records while trees reads them, shows +// as 100%. p.mu must be held. +func (p *progress) percentBar() string { + done := min(float64(p.count)/float64(p.total), 1) + filled := strings.Repeat(barCell, int(done*barWidth)) + + return fmt.Sprintf("%4.0f%% |%-*s|", done*percentScale, barWidth, filled) } // setTotal gives a phase that started without a total its total, once -// it is known. The rate, the ETA and the time with no item completed -// count from then on: the time spent waiting for the total says nothing -// about how fast items go. +// it is known. The rate, the time remaining and the time with no item +// completed count from then on: the time spent waiting for the total +// says nothing about how fast items go. func (p *progress) setTotal(total int64) { if p == nil { return @@ -438,22 +426,26 @@ func (p *progress) finish() { // line formats the phase's progress as of now in at most room columns, // as // -// [3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s +// [3/4] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, remain 13s // -// where "files" is the phase's unit, without the percent when percent -// is false, as before the library's bar, which shows its own. A phase -// with no total shows the items done instead, or what it waits for. The -// rate is that since the first draw in drawn, and the ETA comes from -// the rate since the count started. Once no item has been completed for -// stallTime, the rate, which would read 0, gives way to the word -// "stalled" and where the phase waits, as "stalled disk/db". +// where "files" is the phase's unit. A phase with no total shows the +// items done instead, or what it waits for. The rate is that since the +// first draw in drawn, and the time remaining comes from the rate since +// the count started. Once no item has been completed for stallTime, the +// rate, which would read 0, gives way to the word "stalled" and where +// the phase waits, as "stalled disk/db". // -// The phase and its place, the progress, the elapsed time and the ETA -// are kept whole. A line wider than room leaves out the rate, the words -// of a stalled phase, or what the phase waits for; only if it is still -// wider is it cut at room. Every part is ASCII, so its length is its -// width in columns. p.mu must be held. -func (p *progress) line(now time.Time, room int, percent bool) string { +// When bar is true, as on a terminal, a phase with an exact total ends +// the line with its percent and bar, in place of the percent after its +// count. A total of 0 leaves the bar nothing to fill. +// +// The phase and its place, the progress, the elapsed time and the time +// remaining are kept whole. A line wider than room first leaves out the +// rate, the words of a stalled phase, or what the phase waits for. Next +// it leaves out the bar. Only if it is still wider is it cut at room. +// Every part but the bar is ASCII, so its length is its width in +// columns. p.mu must be held. +func (p *progress) line(now time.Time, room int, bar bool) string { elapsed := now.Sub(p.start).Round(time.Second) // head and tail hold the parts kept whole; extra, between them, is @@ -462,21 +454,41 @@ func (p *progress) line(now time.Time, room int, percent bool) string { if p.total < 0 && p.waitingFor != "" { head, extra = p.label+": ", p.waitingFor+", " - tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed) + tail = fmt.Sprintf("elapsed %s, remain unknown", elapsed) } else { since := p.drawn[0] rate := perSecond(p.count-since.count, now.Sub(since.at)) average := perSecond(p.count, now.Sub(p.rateStart)) - head = p.label + ": " + p.itemsDone(percent) + ", " + head = p.label + ": " + p.itemsDone(true) + ", " extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit) - tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average)) + tail = fmt.Sprintf("elapsed %s, remain %s", elapsed, p.remain(average)) if now.Sub(p.lastItem) >= stallTime { extra = "stalled " + p.stalledOn + ", " } } + if bar && p.total > 0 && !p.estimate { + // Beside the bar, which shows the percent, the count shows none. + // The line is padded to the room the bar leaves it, so the bar + // stays at room's right edge however long the line is. + beside := room - barColumns + barHead := p.label + ": " + p.itemsDone(false) + ", " + + switch { + case len(barHead+extra+tail) <= beside: + return fmt.Sprintf("%-*s%s", beside, barHead+extra+tail, + p.percentBar()) + case len(barHead+tail) <= beside: + return fmt.Sprintf("%-*s%s", beside, barHead+tail, p.percentBar()) + } + + // The bar gives way. The rate, or the words of a stalled phase, + // gave way before it and stay out. + extra = "" + } + full, kept := head+extra+tail, head+tail switch { @@ -526,10 +538,10 @@ func (p *progress) itemsDone(percent bool) string { return fmt.Sprintf("%s %.0f%%", done, pct) } -// eta returns the time the phase has left at rate items a second, or +// remain returns the time the phase has left at rate items a second, or // "unknown": with no total, before the first item, and once the count // passes an estimate. p.mu must be held. -func (p *progress) eta(rate float64) string { +func (p *progress) remain(rate float64) string { switch { case p.total < 0, p.estimate && p.count > p.total: return "unknown" @@ -539,8 +551,8 @@ func (p *progress) eta(rate float64) string { return "unknown" } - remaining := time.Duration( + left := time.Duration( float64(p.total-p.count) / rate * float64(time.Second)) - return remaining.Round(time.Second).String() + return left.Round(time.Second).String() } diff --git a/progress_test.go b/progress_test.go index 1c65053..4093aa6 100644 --- a/progress_test.go +++ b/progress_test.go @@ -23,7 +23,7 @@ const testRedraw = time.Millisecond const testDeadline = 5 * time.Second // hashStalledOn is where the hash phase says it waits once it has -// stalled, as readRuns gives it. +// stalled, as hashPhase gives it. const hashStalledOn = "disk/db" // testClock is a clock a test moves by hand, for a display that reads @@ -117,38 +117,38 @@ func TestProgressLine(t *testing.T) { label: scanPhase("load"), total: -1, waitingFor: countingRecords, }, - want: "[1/5] load: counting the records, elapsed 10s, eta unknown", + want: "[1/4] load: counting the records, elapsed 10s, remain unknown", }, { p: &progress{ label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20, }, - want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s", + want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100}, - want: "[3/5] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown", + want: "[3/4] hash: [0/100] 0%, 0 files/s, elapsed 10s, remain unknown", }, { p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0}, - want: "[4/5] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s", + want: "[4/4] update: [0/0] 100%, 0 files/s, elapsed 10s, remain 0s", }, { p: &progress{ label: scanPhase("walk"), unit: scanUnit, total: 100, estimate: true, count: 20, }, - want: "[2/5] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s", + want: "[2/4] walk: [20/~100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { // Past the estimate the walk cannot tell how long it has left. p: &progress{ label: scanPhase("walk"), unit: scanUnit, total: 100, estimate: true, count: 120, }, - want: "[2/5] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " + - "eta unknown", + want: "[2/4] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " + + "remain unknown", }, { // A first scan's walk has no estimate. p: &progress{ label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20, }, - want: "[2/5] walk: 20 files, 2 files/s, elapsed 10s, eta unknown", + want: "[2/4] walk: 20 files, 2 files/s, elapsed 10s, remain unknown", }, { // The rate is in what the phase counts. p: &progress{ @@ -156,14 +156,14 @@ func TestProgressLine(t *testing.T) { count: 20, }, want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " + - "eta 40s", + "remain 40s", }} for _, tc := range cases { tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now tc.p.drawn = []drawnCount{{at: ago}} - if got := tc.p.line(now, math.MaxInt, true); got != tc.want { + if got := tc.p.line(now, math.MaxInt, false); got != tc.want { t.Errorf("line = %q, want %q", got, tc.want) } } @@ -184,9 +184,9 @@ func TestLineFitsRoom(t *testing.T) { start: ago, rateStart: ago, lastItem: now, drawn: []drawnCount{{at: ago}}, } - content := &progress{ - label: scanPhase("content"), total: -1, - waitingFor: "searching the database", start: ago, rateStart: ago, + load := &progress{ + label: scanPhase("load"), total: -1, + waitingFor: countingRecords, start: ago, rateStart: ago, } started := now.Add(-50 * time.Second) @@ -198,48 +198,49 @@ func TestLineFitsRoom(t *testing.T) { } cases := []struct { - p *progress - room int - percent bool - want string + p *progress + room int + bar bool + want string }{{ - p: hash, room: 57, percent: true, - want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s", + p: hash, room: 60, + want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { - p: hash, room: 56, percent: true, - want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40s", + p: hash, room: 59, + want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40s", }, { - p: hash, room: 45, percent: true, - want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40", + p: hash, room: 48, + want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40", }, { - // Before the library's bar, which shows the percent. - p: hash, room: 57, percent: false, - want: "[3/5] hash: [20/100], 2 files/s, elapsed 10s, eta 40s", + // The bar shows the percent. + p: hash, room: 74, bar: true, + want: "[3/4] hash: [20/100], 2 files/s, elapsed 10s, remain 40s" + + " 20% |██ |", }, { - p: content, room: 40, percent: true, - want: "[5/5] content: elapsed 10s, eta unknown", + p: load, room: 43, + want: "[1/4] load: elapsed 10s, remain unknown", }, { - p: stalled, room: 65, percent: true, - want: "[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " + - "eta 3m20s", + p: stalled, room: 68, + want: "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " + + "remain 3m20s", }, { - p: stalled, room: 64, percent: true, - want: "[3/5] hash: [20/100] 20%, elapsed 50s, eta 3m20s", + p: stalled, room: 67, + want: "[3/4] hash: [20/100] 20%, elapsed 50s, remain 3m20s", }} for _, tc := range cases { - if got := tc.p.line(now, tc.room, tc.percent); got != tc.want { - t.Errorf("line(%d, %t) = %q, want %q", - tc.room, tc.percent, got, tc.want) + if got := tc.p.line(now, tc.room, tc.bar); got != tc.want { + t.Errorf("line(%d, %t) = %q, want %q", tc.room, tc.bar, got, + tc.want) } } } // TestSetTotalStartsRate checks that setTotal starts a phase's rate, -// its ETA and its time with no record read from when the total becomes -// known, not from when the phase started waiting for it: after 20 -// seconds of waiting for the total, the phase has not stalled 5 seconds -// after it, and has 10 seconds after it with no record read. +// its time remaining and its time with no record read from when the +// total becomes known, not from when the phase started waiting for it: +// after 20 seconds of waiting for the total, the phase has not stalled 5 +// seconds after it, and has 10 seconds after it with no record read. func TestSetTotalStartsRate(t *testing.T) { t.Parallel() @@ -255,15 +256,15 @@ func TestSetTotalStartsRate(t *testing.T) { clock.Add(5 * time.Second) - want := "[1/5] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown" - if got := p.line(clock.Now(), math.MaxInt, true); got != want { + want := "[1/4] load: [0/100] 0%, 0 files/s, elapsed 25s, remain unknown" + if got := p.line(clock.Now(), math.MaxInt, false); got != want { t.Errorf("line = %q, want %q", got, want) } clock.Add(5 * time.Second) - want = "[1/5] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown" - if got := p.line(clock.Now(), math.MaxInt, true); got != want { + want = "[1/4] load: [0/100] 0%, stalled db, elapsed 30s, remain unknown" + if got := p.line(clock.Now(), math.MaxInt, false); got != want { t.Errorf("line = %q, want %q", got, want) } @@ -272,8 +273,8 @@ func TestSetTotalStartsRate(t *testing.T) { p.increment() } - want = "[1/5] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s" - if got := p.line(clock.Now(), math.MaxInt, true); got != want { + want = "[1/4] load: [20/100] 20%, 2 files/s, elapsed 30s, remain 40s" + if got := p.line(clock.Now(), math.MaxInt, false); got != want { t.Errorf("line = %q, want %q", got, want) } } @@ -316,9 +317,9 @@ func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) { lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n") - want := "[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown" + want := "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown" if len(lines) != 2 || lines[0] != want || - !strings.HasPrefix(lines[1], "[3/5] hash: [1000/1000] 100%, ") { + !strings.HasPrefix(lines[1], "[3/4] hash: [1000/1000] 100%, ") { t.Errorf("stderr lines = %q, want %q, then the line at 1000", lines, want) } @@ -336,7 +337,7 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) { }, nil, testRedraw, time.Now) awaitOutput(t, stderr, - "[1/5] load: counting the records, elapsed 0s, eta unknown\n", 3) + "[1/4] load: counting the records, elapsed 0s, remain unknown\n", 3) p.finish() } @@ -366,23 +367,23 @@ func TestPlainProgressStalledProducer(t *testing.T) { clock.Add(5 * time.Second) awaitOutput(t, stderr, - "[3/5] hash: [20/100] 20%, 4 files/s, elapsed 5s, eta 20s\n", 3) + "[3/4] hash: [20/100] 20%, 4 files/s, elapsed 5s, remain 20s\n", 3) clock.Add(time.Second) awaitOutput(t, stderr, - "[3/5] hash: [20/100] 20%, 0 files/s, elapsed 6s, eta 24s\n", 3) + "[3/4] hash: [20/100] 20%, 0 files/s, elapsed 6s, remain 24s\n", 3) clock.Add(4 * time.Second) awaitOutput(t, stderr, - "[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n", + "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 10s, remain 40s\n", 3) } // TestPlainProgressRateAfterStall draws a phase's plain line by hand, a // little over plainInterval after the line before, as a late tick // does, through a stall and a burst of files after it: the rate covers -// the files read since the line before, while the ETA comes from the -// phase's average. +// the files read since the line before, while the time remaining comes +// from the phase's average. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestPlainProgressRateAfterStall(t *testing.T) { @@ -410,10 +411,10 @@ func TestPlainProgressRateAfterStall(t *testing.T) { p.finish() want := []string{ - "[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown", - "[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown", - "[3/5] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown", - "[3/5] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s", + "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown", + "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 5s, remain unknown", + "[3/4] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, remain unknown", + "[3/4] hash: [100/1000] 10%, 20 files/s, elapsed 15s, remain 2m15s", } lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n") @@ -426,8 +427,8 @@ func TestPlainProgressRateAfterStall(t *testing.T) { // redrawInterval, as a terminal does, through a stall of stallTime and // a burst of files after it: the rate covers the files read over the // last rateWindow, and falls to 0 once the burst is older than that, -// while the ETA comes from the phase's average. Plain lines show each -// draw. +// while the time remaining comes from the phase's average. Plain lines +// show each draw. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressRateAfterStallRedrawn(t *testing.T) { @@ -457,8 +458,8 @@ func TestProgressRateAfterStallRedrawn(t *testing.T) { redraw(rateWindow + redrawInterval) p.finish() - want := "[3/5] hash: [50/1000] 5%, 10 files/s, elapsed 15s, eta 4m45s\n" + - "[3/5] hash: [50/1000] 5%, 0 files/s, elapsed 15s, eta 4m47s\n" + want := "[3/4] hash: [50/1000] 5%, 10 files/s, elapsed 15s, remain 4m45s\n" + + "[3/4] hash: [50/1000] 5%, 0 files/s, elapsed 15s, remain 4m47s\n" if !strings.Contains(stderr(), want) { t.Errorf("stderr = %q, want the lines %q", stderr(), want) } @@ -480,10 +481,10 @@ func lastFrame(out string) string { } // TestTerminalRedrawsWhileWaiting checks that the terminal display of a -// phase, spinner or bar, keeps being drawn while the phase waits for its -// next item, with the elapsed time as of now, and shows every item -// completed before the wait. stderr is not a terminal here, so the -// displays take it to be fallbackWidth wide. +// phase, with a bar or without, keeps being drawn while the phase waits +// for its next item, with the elapsed time as of now, and shows every +// item completed before the wait. The terminal is 120 columns wide, so +// the hash phase's display has room for its bar. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalRedrawsWhileWaiting(t *testing.T) { @@ -506,7 +507,8 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) { for _, tc := range cases { before := len(stderr()) since := func() string { return stderr()[before:] } - p := startProgress(tc.params, terminalWidth, testRedraw, time.Now) + p := startProgress(tc.params, func() int { return 120 }, testRedraw, + time.Now) for range 50 { p.increment() @@ -552,13 +554,13 @@ func drawLate(stderr func() string, params progressParams, width int, return stderr()[before:] } -// TestTerminalLinesKeepETA checks the line each display shows on an -// 80-column terminal at a large scan's counts and times, and the line -// of trees' read phase at the same counts: the rate is left out, so -// that the elapsed time and the ETA stay whole. +// TestTerminalLinesKeepTimeRemaining checks the line each display shows +// on an 80-column terminal at a large scan's counts and times, and the +// line of trees' read phase at the same counts: the rate is left out, +// so that the elapsed time and the time remaining stay whole. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr -func TestTerminalLinesKeepETA(t *testing.T) { +func TestTerminalLinesKeepTimeRemaining(t *testing.T) { stderr := captureStderr(t) cases := []struct { @@ -570,26 +572,20 @@ func TestTerminalLinesKeepETA(t *testing.T) { label: scanPhase("walk"), unit: scanUnit, total: 1200000, estimate: true, }, - want: "[2/5] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " + - "eta unknown", + want: "[2/4] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " + + "remain unknown", }, { params: progressParams{ label: scanPhase("hash"), unit: scanUnit, total: 9876543, }, - want: "[3/5] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " + - "eta 17h30m0s", - }, { - params: progressParams{ - label: scanPhase("content"), unit: scanUnit, total: 9876543, - }, - want: "[5/5] content: [1234567/9876543] 12%, elapsed 2h30m0s, " + - "eta 17h30m0s", + want: "[3/4] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " + + "remain 17h30m0s", }, { params: progressParams{ label: treesPhase("read"), unit: "records", total: 9876543, }, want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " + - "eta 17h30m0s", + "remain 17h30m0s", }} for _, tc := range cases { @@ -600,30 +596,32 @@ func TestTerminalLinesKeepETA(t *testing.T) { } } -// TestTerminalStalledLines draws every phase's line on an 80-column -// terminal once the phase has stalled, its last item done 42 seconds -// before. At an ordinary scan's counts and times, 12345 of 45678 items -// done four and a half minutes in, the line shows where the phase waits -// whole, beside the phase, progress, elapsed time and ETA. At a large -// scan's, 1234567 of 9876543 two and a half hours in, those four still -// show whole, as the words give way first where they do not fit. +// TestTerminalStalledLines draws every phase's line once the phase has +// stalled, its last item done 42 seconds before. At an ordinary scan's +// counts and times, 12345 of 45678 items done four and a half minutes +// in, on a terminal 100 columns wide, the line shows where the phase +// waits whole, beside the phase, progress, elapsed time and time +// remaining. At a large scan's, 1234567 of 9876543 two and a half hours +// in, on an 80-column terminal, those four still show whole, as the +// words give way first where they do not fit. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalStalledLines(t *testing.T) { stderr := captureStderr(t) - // drawStalled draws the display params describes as it starts and - // once more as it ends, with count of total items done elapsed into - // the phase, and returns what the terminal shows. - drawStalled := func(params progressParams, total, count int64, - elapsed time.Duration, + // drawStalled draws the display params describes on a terminal width + // columns wide as it starts and once more as it ends, with count of + // total items done elapsed into the phase, and returns what the + // terminal shows. + drawStalled := func(params progressParams, width int, total, + count int64, elapsed time.Duration, ) string { before := len(stderr()) clock := &testClock{now: time.Now()} params.total = total // No timed draws come between the two. - p := startProgress(params, func() int { return 80 }, time.Hour, + p := startProgress(params, func() int { return width }, time.Hour, clock.Now) clock.Add(elapsed - 42*time.Second) @@ -643,9 +641,6 @@ func TestTerminalStalledLines(t *testing.T) { {label: scanPhase("walk"), estimate: true, stalledOn: "disk"}, {label: scanPhase("hash"), stalledOn: hashStalledOn}, {label: scanPhase("update"), stalledOn: "db"}, - // The content phase's checks and reads, then its last write. - {label: scanPhase("content"), stalledOn: "disk/db"}, - {label: scanPhase("content"), stalledOn: "db"}, {label: treesPhase("read"), stalledOn: "db"}, {label: treesPhase("group"), stalledOn: "cpu"}, {label: treesPhase("write"), stalledOn: "out"}, @@ -658,19 +653,22 @@ func TestTerminalStalledLines(t *testing.T) { approx = "~" } - shown := drawStalled(params, 45678, 12345, + // The percent is in the line, or beside the bar. + shown := drawStalled(params, 100, 45678, 12345, 4*time.Minute+30*time.Second) - want := params.label + ": [12345/" + approx + "45678] 27%, stalled " + - params.stalledOn + ", elapsed 4m30s, eta 12m9s" - - if !strings.Contains(shown, want) { - t.Errorf("terminal shows %q, want %q", shown, want) + for _, want := range []string{ + params.label + ": [12345/" + approx + "45678]", + ", stalled " + params.stalledOn + ", elapsed 4m30s, remain 12m9s", + } { + if !strings.Contains(shown, want) { + t.Errorf("terminal shows %q, want %q", shown, want) + } } - shown = drawStalled(params, 9876543, 1234567, 150*time.Minute) + shown = drawStalled(params, 80, 9876543, 1234567, 150*time.Minute) for _, want := range []string{ params.label + ": [1234567/" + approx + "9876543] 12%, ", - ", elapsed 2h30m0s, eta 17h30m0s", + ", elapsed 2h30m0s, remain 17h30m0s", } { if !strings.Contains(shown, want) { t.Errorf("terminal shows %q, want %q", shown, want) @@ -693,7 +691,7 @@ func TestTerminalLinesFitWidth(t *testing.T) { for _, width := range []int{120, 100, 80, 72, 60, 50, 40} { for _, total := range []int64{9876543, 20000} { params := progressParams{ - label: scanPhase("content"), unit: scanUnit, total: total, + label: scanPhase("update"), unit: scanUnit, total: total, } out := drawLate(stderr, params, width, total/8) @@ -715,13 +713,164 @@ func TestTerminalLinesFitWidth(t *testing.T) { } } +// TestTerminalLinesStartWithSpinner draws a display with no total and +// one with a bar, two and a half hours into the phase, on a terminal 120 +// columns wide. Each line must start with the library's spinner in the +// first column, then a space and the phase's label. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalLinesStartWithSpinner(t *testing.T) { + stderr := captureStderr(t) + + for _, params := range []progressParams{ + {label: scanPhase("walk"), unit: scanUnit, total: -1}, + {label: scanPhase("hash"), unit: scanUnit, total: 9876543}, + } { + shown := lastFrame(drawLate(stderr, params, 120, 1234567)) + spinner, rest, _ := strings.Cut(shown, " ") + + if utf8.RuneCountInString(spinner) != 1 || + !strings.HasPrefix(rest, params.label+": ") { + t.Errorf("terminal shows %q, want the spinner, a space and %q", + shown, params.label) + } + + if params.total > 0 && !strings.Contains(shown, "% |") { + t.Errorf("terminal shows %q, want a bar", shown) + } + } +} + +// TestTerminalBarStaysAtRightEdge draws the hash phase's display two +// and a half hours in, at two counts whose lines differ in length, on +// terminals 110 and 120 columns wide. The bar is partly filled at the +// first count and full at the second. It must start the same number of +// columns from the terminal's right edge each time, and end the same +// number of columns from it: a line that grows or shrinks, as its time +// remaining does, never moves the bar, and nor do the bar's filled +// cells. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalBarStaysAtRightEdge(t *testing.T) { + stderr := captureStderr(t) + params := progressParams{ + label: scanPhase("hash"), unit: scanUnit, total: 9876543, + } + + widths := []int{110, 120} + counts := []int64{1234567, 9876543} + startFromEdge := make([]int, 0, len(widths)*len(counts)) + endFromEdge := make([]int, 0, len(widths)*len(counts)) + + for _, width := range widths { + for _, count := range counts { + shown := lastFrame(drawLate(stderr, params, width, count)) + + beforeBar, _, found := strings.Cut(shown, "% |") + if !found { + t.Fatalf("terminal %d wide shows %q, want a bar", width, shown) + } + + // The bar ends with the last "|". + throughBar := shown[:strings.LastIndex(shown, "|")+1] + + startFromEdge = append(startFromEdge, + width-utf8.RuneCountInString(beforeBar)) + endFromEdge = append(endFromEdge, + width-utf8.RuneCountInString(throughBar)) + } + } + + if slices.Min(startFromEdge) != slices.Max(startFromEdge) { + t.Errorf("bar starts %v columns from the right edge, want the same "+ + "every time", startFromEdge) + } + + if slices.Min(endFromEdge) != slices.Max(endFromEdge) { + t.Errorf("bar ends %v columns from the right edge, want the same "+ + "every time", endFromEdge) + } +} + +// TestTerminalRateGivesWayBeforeBar draws the hash phase's display two +// and a half hours in. On a terminal 90 columns wide the line fits +// beside the bar without its rate, but not with it: the bar is drawn +// and the rate is not. On a terminal 85 columns wide the line does not +// fit beside the bar even without its rate: neither is drawn, though +// the rate would fit where the bar was. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalRateGivesWayBeforeBar(t *testing.T) { + stderr := captureStderr(t) + params := progressParams{ + label: scanPhase("hash"), unit: scanUnit, total: 9876543, + } + + shown := lastFrame(drawLate(stderr, params, 90, 1234567)) + if !strings.Contains(shown, "% |") || strings.Contains(shown, "files/s") { + t.Errorf("terminal 90 wide shows %q, want the bar and no rate", shown) + } + + shown = lastFrame(drawLate(stderr, params, 85, 1234567)) + if strings.Contains(shown, "% |") || strings.Contains(shown, "files/s") { + t.Errorf("terminal 85 wide shows %q, want neither bar nor rate", shown) + } +} + +// TestTerminalSpinnerTurnsBesideBar draws the hash phase's display, its +// bar partly filled, several times on a terminal 120 columns wide. The +// spinner must turn at every draw. A filled cell of the bar takes one +// column but three bytes. A line measured in bytes would seem too wide +// for the terminal, and every draw would start the display over, its +// spinner included. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalSpinnerTurnsBesideBar(t *testing.T) { + stderr := captureStderr(t) + params := progressParams{ + label: scanPhase("hash"), unit: scanUnit, total: 9876543, + } + + // No timed draws: the test draws. + p := startProgress(params, func() int { return 120 }, time.Hour, + time.Now) + + p.mu.Lock() + p.count = 1234567 + p.mu.Unlock() + + const draws = 4 + + spinners := make([]string, 0, draws) + + for range draws { + before := len(stderr()) + + p.draw() + + spinner, _, _ := strings.Cut(lastFrame(stderr()[before:]), " ") + spinners = append(spinners, spinner) + } + + p.finish() + + for i := 1; i < len(spinners); i++ { + if spinners[i] == spinners[i-1] { + t.Errorf("spinners at successive draws are %q, want each to "+ + "differ from the one before", spinners) + + break + } + } +} + // TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two // and a half hours into the phase, on a terminal 120 columns wide, then // makes the terminal narrower than that line and draws it again a few // times. Nothing written after that, the erase before each redraw // included, may reach the narrower terminal's last column: it would -// wrap, and every redraw would leave a line behind. At these widths the -// spinner stays a spinner and the bar gives way to the spinner. +// wrap, and every redraw would leave a line behind. At the narrower +// width the hash phase's bar gives way. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { @@ -739,7 +888,7 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { progressParams{ label: scanPhase("hash"), unit: scanUnit, total: 9876543, }, - 90, + 80, }, } @@ -779,10 +928,10 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { } } -// TestProgressWarningsOnOwnLines drives the terminal displays, spinner -// and bar, through a run of warnings with no items between them, as when -// the walk meets many unreadable paths, for many redraws: every warning -// must land on a line of its own, never inside a redraw. +// TestProgressWarningsOnOwnLines drives the terminal displays, with a +// bar and without, through a run of warnings with no items between +// them, as when the walk meets many unreadable paths, for many redraws: +// every warning must land on a line of its own, never inside a redraw. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressWarningsOnOwnLines(t *testing.T) { @@ -828,11 +977,12 @@ func TestProgressWarningsOnOwnLines(t *testing.T) { } } -// TestProgressWriterRowsOnOwnLines drives the terminal displays, -// spinner and bar, through a run of rows written through the display's +// TestProgressWriterRowsOnOwnLines drives the terminal displays, with a +// bar and without, through a run of rows written through the display's // writer to the same terminal, as stdout is in a plain "sfdupes trees", // each written out at once, for many redraws: the terminal must show -// every row whole, in order, on a line of its own. +// every row whole, in order, on a line of its own. The terminal is 120 +// columns wide, so the write phase's display has room for its bar. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressWriterRowsOnOwnLines(t *testing.T) { @@ -849,7 +999,8 @@ func TestProgressWriterRowsOnOwnLines(t *testing.T) { var want []string for _, params := range cases { - p := startProgress(params, terminalWidth, testRedraw, time.Now) + p := startProgress(params, func() int { return 120 }, testRedraw, + time.Now) out := p.writer(os.Stderr) // No pause between rows: one is garbled only if a redraw lands diff --git a/report.go b/report.go index a3396fe..5db1217 100644 --- a/report.go +++ b/report.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "fmt" "io" @@ -24,9 +25,9 @@ const minGroupSize = 2 type scanRec struct { size int64 mtime time.Time - head string - tail string - content string + head [sha256.Size]byte + tail [sha256.Size]byte + content [sha256.Size]byte path string } @@ -169,12 +170,6 @@ type reportWriter struct { func (w *reportWriter) add(r scanRec) error { w.records++ - // A record without a content hash has unknown content (README - // "Database"), so it is never a duplicate. - if r.content == "" { - return nil - } - if r.size == w.sig.size && r.head == w.sig.head && r.tail == w.sig.tail && r.content == w.sig.content { w.paths = append(w.paths, r.path) diff --git a/report_test.go b/report_test.go index bb27652..9b47034 100644 --- a/report_test.go +++ b/report_test.go @@ -2,6 +2,7 @@ package main import ( "bytes" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -18,14 +19,27 @@ import ( // awkwardDir is a directory name holding every byte the reports escape. const awkwardDir = "/d/\tone\ntwo\rthree\\four" +// testHash returns a stand-in hash for a test record: the bytes of s, +// then zeros. Stand-ins sort as their strings do, so a test can choose +// the order in which report reads its records. +func testHash(s string) [sha256.Size]byte { + var h [sha256.Size]byte + + copy(h[:], s) + + return h +} + // awkwardPairRecs is a duplicate pair in sibling directories /d/A and // awkwardDir. A raw tab sorts before "A" but its escaped form `\t` // sorts after it, so awkwardDir coming first shows that sorting uses // the raw path. func awkwardPairRecs() []scanRec { + h, tl, c := testHash("h"), testHash("t"), testHash("c") + return []scanRec{ - {size: 5, head: "h", tail: "t", content: "c", path: "/d/A/f"}, - {size: 5, head: "h", tail: "t", content: "c", path: awkwardDir + "/f"}, + {size: 5, head: h, tail: tl, content: c, path: "/d/A/f"}, + {size: 5, head: h, tail: tl, content: c, path: awkwardDir + "/f"}, } } @@ -143,8 +157,8 @@ func TestReportStdoutFailsWhileReading(t *testing.T) { recs := make([]scanRec, 2*ioBufSize/len(dir)) for i := range recs { recs[i] = scanRec{ - size: int64(i/2) + 1, head: "h", tail: "t", content: "c", - path: fmt.Sprintf("%s/%d", dir, i), + size: int64(i/2) + 1, head: testHash("h"), tail: testHash("t"), + content: testHash("c"), path: fmt.Sprintf("%s/%d", dir, i), } } @@ -173,7 +187,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) { // reportOpenLine is the plain line of report's open phase when it opens // the database at once; it prints it as the phase starts and as it ends. const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" // smokeReport is report's stdout for the smoke-test records. const smokeReport = "first\tdupe\tsize\n" + @@ -211,7 +225,7 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) { "report: database " + path + " (from " + databaseEnv + ")", reportOpenLine, reportOpenLine, - "[2/2] read: counting the records, elapsed 0s, eta unknown", + "[2/2] read: counting the records, elapsed 0s, remain unknown", "[2/2] read: [6/6] 100%, ", "report: 6 records read, 2 duplicate groups, 4 dupe files, " + "6.1 KiB reclaimable", @@ -277,13 +291,14 @@ func TestReadReportOnTerminal(t *testing.T) { func TestRunReportsIgnoreInsertionOrder(t *testing.T) { // README §Constraints: identical database contents give identical // output, whatever order the records were inserted in. + a, b := testHash("a"), testHash("b") + recs := append(smokeTreeRecs(), awkwardPairRecs()...) recs = append(recs, - scanRec{size: 50, head: "b", tail: "b", content: "b", path: "/y/2"}, - scanRec{size: 50, head: "b", tail: "b", content: "b", path: "/y/1"}, - scanRec{size: 50, head: "a", tail: "a", content: "a", path: "/x/2"}, - scanRec{size: 50, head: "a", tail: "a", content: "a", path: "/x/1"}, - scanRec{size: 50, path: "/x/unhashed"}, + scanRec{size: 50, head: b, tail: b, content: b, path: "/y/2"}, + scanRec{size: 50, head: b, tail: b, content: b, path: "/y/1"}, + scanRec{size: 50, head: a, tail: a, content: a, path: "/x/2"}, + scanRec{size: 50, head: a, tail: a, content: a, path: "/x/1"}, ) reversed := slices.Clone(recs) @@ -386,16 +401,19 @@ func TestWarnfEscapes(t *testing.T) { func TestDupeGroups(t *testing.T) { t.Parallel() + h, tl, c := testHash("h"), testHash("t"), testHash("c") + bigH, bigT, bigC := testHash("H"), testHash("T"), testHash("C") + u := testHash("u") recs := []scanRec{ - {size: 100, head: "h", tail: "t", content: "c", path: "/z/b"}, - {size: 100, head: "h", tail: "t", content: "c", path: "/z/a"}, - {size: 100, head: "h", tail: "t", content: "c", path: "/z/c"}, - {size: 4000, head: "H", tail: "T", content: "C", path: "/big/2"}, - {size: 4000, head: "H", tail: "T", content: "C", path: "/big/1"}, + {size: 100, head: h, tail: tl, content: c, path: "/z/b"}, + {size: 100, head: h, tail: tl, content: c, path: "/z/a"}, + {size: 100, head: h, tail: tl, content: c, path: "/z/c"}, + {size: 4000, head: bigH, tail: bigT, content: bigC, path: "/big/2"}, + {size: 4000, head: bigH, tail: bigT, content: bigC, path: "/big/1"}, // Same size as the /z group but a different head hash. - {size: 100, head: "other", tail: "t", content: "c", path: "/z/d"}, + {size: 100, head: testHash("other"), tail: tl, content: c, path: "/z/d"}, // A singleton signature must not form a group. - {size: 7, head: "u", tail: "u", content: "u", path: "/lonely"}, + {size: 7, head: u, tail: u, content: u, path: "/lonely"}, } groups := dupeGroupsOf(t, recs) @@ -419,16 +437,13 @@ func TestDupeGroups(t *testing.T) { func TestDupeGroupsContentSeparates(t *testing.T) { t.Parallel() - // Same size, head, and tail, but different content hashes: the final - // rung keeps them apart, so no group forms. Matching content groups. - // Records without a content hash never group, not even with each - // other. + // Same size, head, and tail: /b differs from the others only in its + // content hash, so it joins no group, while /a and /c group. + h, tl := testHash("h"), testHash("t") recs := []scanRec{ - {size: 100, head: "h", tail: "t", content: "c1", path: "/a"}, - {size: 100, head: "h", tail: "t", content: "c2", path: "/b"}, - {size: 100, head: "h", tail: "t", content: "c1", path: "/c"}, - {size: 100, head: "h", tail: "t", path: "/d"}, - {size: 100, head: "h", tail: "t", path: "/e"}, + {size: 100, head: h, tail: tl, content: testHash("c1"), path: "/a"}, + {size: 100, head: h, tail: tl, content: testHash("c2"), path: "/b"}, + {size: 100, head: h, tail: tl, content: testHash("c1"), path: "/c"}, } groups := dupeGroupsOf(t, recs) @@ -447,14 +462,15 @@ func TestDupeGroupsMtimeExcluded(t *testing.T) { // mtime is informational only; records differing only in mtime // still group together. + h, tl, c := testHash("h"), testHash("t"), testHash("c") recs := []scanRec{ { - size: 9, mtime: time.Unix(100, 0), head: "h", tail: "t", - content: "c", path: "/m/1", + size: 9, mtime: time.Unix(100, 0), head: h, tail: tl, + content: c, path: "/m/1", }, { - size: 9, mtime: time.Unix(200, 0), head: "h", tail: "t", - content: "c", path: "/m/2", + size: 9, mtime: time.Unix(200, 0), head: h, tail: tl, + content: c, path: "/m/2", }, } @@ -469,11 +485,12 @@ func TestDupeGroupsTieBreak(t *testing.T) { // The hashes sort opposite to the first paths, so ordering the // groups by hash instead of by first path fails this test. + a, b := testHash("a"), testHash("b") recs := []scanRec{ - {size: 50, head: "a", tail: "a", content: "a", path: "/beta/2"}, - {size: 50, head: "a", tail: "a", content: "a", path: "/beta/1"}, - {size: 50, head: "b", tail: "b", content: "b", path: "/alpha/2"}, - {size: 50, head: "b", tail: "b", content: "b", path: "/alpha/1"}, + {size: 50, head: a, tail: a, content: a, path: "/beta/2"}, + {size: 50, head: a, tail: a, content: a, path: "/beta/1"}, + {size: 50, head: b, tail: b, content: b, path: "/alpha/2"}, + {size: 50, head: b, tail: b, content: b, path: "/alpha/1"}, } groups := dupeGroupsOf(t, recs) @@ -495,15 +512,16 @@ func TestDupeGroupsOrderedByFirstPath(t *testing.T) { // /z, /k, which is neither the order of their first paths nor its // reverse, and must write them ordered by first path. The group of // size 7 comes after them all, though its first path sorts first. + a, b, c, d := testHash("a"), testHash("b"), testHash("c"), testHash("d") recs := []scanRec{ - {size: 50, head: "a", tail: "a", content: "a", path: "/k/2"}, - {size: 50, head: "a", tail: "a", content: "a", path: "/k/1"}, - {size: 50, head: "b", tail: "b", content: "b", path: "/z/1"}, - {size: 50, head: "b", tail: "b", content: "b", path: "/z/2"}, - {size: 50, head: "c", tail: "c", content: "c", path: "/a/2"}, - {size: 50, head: "c", tail: "c", content: "c", path: "/a/1"}, - {size: 7, head: "d", tail: "d", content: "d", path: "/0/2"}, - {size: 7, head: "d", tail: "d", content: "d", path: "/0/1"}, + {size: 50, head: a, tail: a, content: a, path: "/k/2"}, + {size: 50, head: a, tail: a, content: a, path: "/k/1"}, + {size: 50, head: b, tail: b, content: b, path: "/z/1"}, + {size: 50, head: b, tail: b, content: b, path: "/z/2"}, + {size: 50, head: c, tail: c, content: c, path: "/a/2"}, + {size: 50, head: c, tail: c, content: c, path: "/a/1"}, + {size: 7, head: d, tail: d, content: d, path: "/0/2"}, + {size: 7, head: d, tail: d, content: d, path: "/0/1"}, } groups := dupeGroupsOf(t, recs) @@ -522,11 +540,12 @@ func TestDupeGroupsOrderedByFirstPath(t *testing.T) { func TestDupeGroupsDeterministic(t *testing.T) { t.Parallel() + a, b := testHash("a"), testHash("b") recs := []scanRec{ - {size: 1, head: "a", tail: "a", content: "a", path: "/p/1"}, - {size: 1, head: "a", tail: "a", content: "a", path: "/p/2"}, - {size: 2, head: "b", tail: "b", content: "b", path: "/q/1"}, - {size: 2, head: "b", tail: "b", content: "b", path: "/q/2"}, + {size: 1, head: a, tail: a, content: a, path: "/p/1"}, + {size: 1, head: a, tail: a, content: a, path: "/p/2"}, + {size: 2, head: b, tail: b, content: b, path: "/q/1"}, + {size: 2, head: b, tail: b, content: b, path: "/q/2"}, } forward := dupeGroupsOf(t, recs) diff --git a/scan.go b/scan.go index 587276f..a16dd84 100644 --- a/scan.go +++ b/scan.go @@ -5,7 +5,6 @@ import ( "context" "crypto/sha256" "database/sql" - "encoding/hex" "errors" "fmt" "io" @@ -21,32 +20,21 @@ import ( "time" ) -// The duplicate ladder (see hashSignature and README "Duplicate -// detection"). A same-size candidate below headTailMin is hashed in -// full and compared directly; a larger one is separated first by the -// hashes of its end windows, then by a content hash that is exact below -// wholeFileMax and deliberately sampled at or above it. The hash phase -// reads only the end windows of a larger file; the content phase reads -// it for its content hash only once its size, head, and tail match -// another file's. - -// headTailMin is the size threshold for the end-window gate. A file -// smaller than this is hashed in full directly, with no separate head -// and tail step: its head, tail, and content all carry the whole-file -// hash. A file this size or larger is separated first by its end -// windows. -const headTailMin = 10 * 1024 * 1024 +// The hash phase gives every file it reads three hashes (see +// hashSignature and README "Duplicate detection"): head and tail, of +// its first and last headTailWindow bytes, and content. The content +// hash covers the whole file below wholeFileMax, and only sampled +// windows at or above it, so such a file is never read in full. // headTailWindow is the number of bytes hashed from each end of a file -// at or above headTailMin (the head and tail rungs). Because -// headTailMin is far larger than two windows, the head and tail windows -// never overlap. +// for its head and tail. In a file no larger than this, each window is +// the whole file. const headTailWindow = 64 * 1024 -// wholeFileMax is the size boundary between the two content rungs: a +// wholeFileMax is the size boundary between the two content hashes: a // file strictly smaller than this is content-hashed in full; a file // this size or larger is content-hashed by sampling. -const wholeFileMax = 50 * 1024 * 1024 +const wholeFileMax = 100 * 1024 * 1024 // sampleStride is the spacing between content samples for large files: // one window is read at each gigabyte-aligned offset (0, 1 GiB, ...). @@ -76,13 +64,11 @@ type fileRec struct { } // fileMeta is the in-memory index entry for one existing database -// record: just enough for change detection, plus whether the record -// carries hashes. Hashes stay on disk; at tens of millions of records -// they would dominate the scan's memory. +// record: just enough for change detection. Hashes stay on disk; at +// tens of millions of records they would dominate the scan's memory. type fileMeta struct { - size int64 - mtime time.Time - hashed bool + size int64 + mtime time.Time } // scanUnit is what every scan phase counts, as its progress display @@ -90,36 +76,31 @@ type fileMeta struct { const scanUnit = "files" // scanPhase names one phase of a scan with its place among all of them, -// as "[2/5] walk", for the phase's progress display. The list here is +// as "[2/4] walk", for the phase's progress display. The list here is // the one place the phases are counted. func scanPhase(name string) string { - return phaseLabel( - []string{"load", "walk", "hash", "update", "content"}, name) + return phaseLabel([]string{"load", "walk", "hash", "update"}, name) } -// runScan implements the scan subcommand: five sequential phases — +// runScan implements the scan subcommand: four sequential phases — // load (which reads the existing records), walk (which stats each file -// as it is discovered), hash, update, content — that synchronize the -// persistent database with the filesystem state under the PATH -// operands. Only files whose size at least one other file shares are -// ever hashed: a size-unique file cannot be a duplicate. A file of -// headTailMin or more gets its content hash only when its size, head, -// and tail match another file's. Flag parsing and the -// at-least-one-operand check are done by cobra. Before anything else, -// the scan names the database on stderr, saying whether it is new. The -// scan holds the lock on the database for its whole run, so a second -// scan fails before it walks the filesystem or opens the database. -// Errors are returned rather than exiting, so that the deferred close — -// which takes the database out of WAL mode — always runs, and the lock -// is released after it. When ctx is cancelled, as by the SIGINT or -// SIGTERM that interruptContext catches, the scan keeps what it has -// hashed (see syncScan), prints how many files its walk reached, and -// returns errInterrupted. workers must be at least 1; the scan command -// rejects anything less. Before the lock is taken, the scan lists the -// filesystems that are not persistent, on which the walk records -// nothing; unless noDefaultExcludes is set, the walk does not enter the -// default excludes either. Once the database is open, the scan notes -// its files, which the walk passes over. +// as it is discovered), hash, update — that synchronize the persistent +// database with the filesystem state under the PATH operands. Flag +// parsing and the at-least-one-operand check are done by cobra. Before +// anything else, the scan names the database on stderr, saying whether +// it is new. The scan holds the lock on the database for its whole run, +// so a second scan fails before it walks the filesystem or opens the +// database. Errors are returned rather than exiting, so that the +// deferred close — which takes the database out of WAL mode — always +// runs, and the lock is released after it. When ctx is cancelled, as by +// the SIGINT or SIGTERM that interruptContext catches, the scan keeps +// what it has hashed (see syncScan), prints how many files its walk +// reached, and returns errInterrupted. workers must be at least 1; the +// scan command rejects anything less. Before the lock is taken, the +// scan lists the filesystems that are not persistent, on which the walk +// records nothing; unless noDefaultExcludes is set, the walk does not +// enter the default excludes either. Once the database is open, the +// scan notes its files, which the walk passes over. func runScan(ctx context.Context, roots []string, workers int, oneFS, noDefaultExcludes bool, ) error { @@ -358,13 +339,11 @@ type scanStats struct { // whatever remains after the walk and hash phases is deleted by the // update phase. type scanState struct { - db *sql.DB - existing map[string]fileMeta - sizes []int64 // size census: every walked file, plus records outside the roots - toHash []fileRec // files whose size is shared: must be read - sentinels []fileRec // new/changed size-unique files: recorded without hashes - batch []scanRec - st scanStats + db *sql.DB + existing map[string]fileMeta + toHash []fileRec // new and changed files: must be read + batch []scanRec + st scanStats } // syncScan synchronizes the database with the filesystem under roots; @@ -391,25 +370,21 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string, } // runPhases synchronizes the database with the filesystem under roots -// in five sequential phases: load (read the existing records), walk -// (enumerate and stat every file, building a complete size census), -// hash (read only the new or changed — or previously unhashed — files -// whose size at least one other file shares, committing results in -// batches as they arrive), update (record the size-unique files without -// reading them, and delete the records the scan no longer verifies), -// and content (fill in the content hash of every record of headTailMin -// or more whose size, head, and tail match another record's). Records -// outside the roots are never touched, except that the content phase -// fills in their content hash. Operands the walk cannot start from, or -// that are not on a persistent filesystem, are dropped first, so the -// records beneath them are treated as if those operands had not been -// given. Below an operand, the walk follows rules. +// in four sequential phases: load (read the existing records), walk +// (enumerate and stat every file), hash (read only the new or changed +// files, committing results in batches as they arrive), and update +// (delete the records the scan no longer verifies). Records outside the +// roots are never touched. Operands the walk cannot start from, or that +// are not on a persistent filesystem, are dropped first, so the records +// beneath them are treated as if those operands had not been given. +// Below an operand, the walk follows rules. func (s *scanState) runPhases(ctx context.Context, roots []string, workers int, oneFS bool, rules walkRules, ) error { // Types and filesystems are checked before pruning so that an // operand under a dropped one is still scanned, not dropped as lying - // under it. + // under it, and so that an operand inside a .zfs directory gets its + // warning even when it lies under another operand. roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules, deviceOfPath) @@ -418,41 +393,34 @@ func (s *scanState) runPhases(ctx context.Context, roots []string, return err } - changed, unhashed := s.walkPhase( - startWalk(ctx, roots, oneFS, rules, workers), roots) + s.walkPhase(startWalk(ctx, roots, oneFS, rules, workers), roots) - // A cancelled walk stops early, so its size census covers only part - // of the roots, and every file it never reached would look vanished - // to the update phase. Stop before anything is written or deleted. + // A cancelled walk stops early, so every file it never reached would + // look vanished to the update phase. Stop before anything is written + // or deleted. err = ctx.Err() if err != nil { return err } - s.partition(changed, unhashed) - err = s.hashPhase(ctx, workers) if err != nil { return err } - err = s.updatePhase(ctx) - if err != nil { - return err - } - - return s.contentPhase(ctx, workers, rules) + return s.updatePhase(ctx) } // walkableRoots returns the operands the walk can start from: regular -// files, and directories not named .zfs. Every other operand is warned -// about, counted as skipped, and dropped. A dropped operand is no -// longer a root, so the records stored beneath it are treated as if it -// had not been given: those under no other root count as outside the -// roots and are not deleted as unverified. An operand on a filesystem -// in rules that is not persistent is dropped the same way, with a line -// saying so that is not a warning, and is not counted as skipped. An -// operand that fails lstat here is kept, and the walk warns about it. +// files, and directories not named .zfs, none of them inside a +// directory named .zfs. Every other operand is warned about, counted as +// skipped, and dropped. A dropped operand is no longer a root, so the +// records stored beneath it are treated as if it had not been given: +// those under no other root count as outside the roots and are not +// deleted as unverified. An operand on a filesystem in rules that is not +// persistent is dropped the same way, with a line saying so that is not +// a warning, and is not counted as skipped. An operand that fails lstat +// here is kept, and the walk warns about it unless it no longer exists. func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { kept := make([]string, 0, len(roots)) @@ -483,10 +451,8 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { } // loadIndex indexes the database records under the scan roots for -// change detection and collects the sizes of every record outside -// them: out-of-scope records join the size census so a scanned file -// can be recognized as a possible duplicate of a tree scanned -// separately into the same database. +// change detection. It reads every record, those outside the roots +// included, so its display counts against the number of records. func (s *scanState) loadIndex(ctx context.Context, roots []string) error { // Indexing tens of millions of records takes real time; without a // display the scan looks hung before the walk begins. Counting them @@ -507,36 +473,24 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error { s.existing = make(map[string]fileMeta) return loadFileMeta(ctx, s.db, - func(path string, size int64, mtime time.Time, hashed bool) { + func(path string, size int64, mtime time.Time) { prog.increment() if underAnyRoot(path, roots) { - s.existing[path] = fileMeta{ - size: size, mtime: mtime, hashed: hashed, - } - - return + s.existing[path] = fileMeta{size: size, mtime: mtime} } - - s.sizes = append(s.sizes, size) }) } -// walkPhase drains the walk, appending every walked file's size to -// the census and resolving what it can immediately: an unchanged file -// whose record already has hashes needs nothing from the hash phase -// (the content phase may still fill in its content hash). It returns -// the new-or-changed files and the unchanged files whose records lack -// hashes; both remain candidates until the census decides whether -// their sizes are shared. A record at or below a path the walk left -// out, because -x stopped at it, it is a default exclude, or it is not -// on a persistent filesystem, counts as outside the roots: it is not -// deleted, and its size joins the census. That does not hold below a -// root at or inside that path, since that root's own walk verifies the -// records below it; nor is a line printed for such a root. -func (s *scanState) walkPhase( - events <-chan walkEvent, roots []string, -) ([]fileRec, []fileRec) { +// walkPhase drains the walk and queues every new or changed file for +// the hash phase. An unchanged file needs nothing more, and its record +// counts as verified. A record at or below a path the walk left out, +// because -x stopped at it, it is a default exclude, or it is not on a +// persistent filesystem, counts as outside the roots: it is not +// deleted. That does not hold below a root at or inside that path, +// since that root's own walk verifies the records below it; nor is a +// line printed for such a root. +func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) { // The records the last scan left under the roots estimate how many // files the walk will find. A first scan has none, and no estimate. estimate := int64(len(s.existing)) @@ -549,10 +503,7 @@ func (s *scanState) walkPhase( estimate: true, stalledOn: "disk", }) - var ( - changed, unhashed []fileRec - leftOut []string - ) + var leftOut []string for ev := range events { if ev.otherFSDir != "" { @@ -582,28 +533,25 @@ func (s *scanState) walkPhase( continue } - s.sizes = append(s.sizes, ev.rec.size) s.st.walked++ prog.increment() old, ok := s.existing[ev.rec.path] + if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) { + s.toHash = append(s.toHash, ev.rec) - switch { - case !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime): - changed = append(changed, ev.rec) - case old.hashed: - delete(s.existing, ev.rec.path) - - s.st.unchanged++ - default: - unhashed = append(unhashed, ev.rec) + continue } + + delete(s.existing, ev.rec.path) + + s.st.unchanged++ } prog.finish() - for path, old := range s.existing { + for path := range s.existing { // Kept only when a path the walk left out lies below the deepest // root above the record, or is the record's own. A root at or // below that path was walked on its own and did not find the @@ -612,56 +560,8 @@ func (s *scanState) walkPhase( left := deepestRoot(path, leftOut) if len(left) > len(deepestRoot(path, roots)) { delete(s.existing, path) - - s.sizes = append(s.sizes, old.size) } } - - return changed, unhashed -} - -// partition decides each candidate's disposition now that the size -// census is complete. A file whose size no other file shares cannot -// be a duplicate and is never read: a new or changed one is recorded -// without hashes, an unchanged unhashed one keeps its record. Every -// file with a shared size queues for the hash phase. -func (s *scanState) partition(changed, unhashed []fileRec) { - slices.Sort(s.sizes) - - for _, rec := range changed { - if s.sizeShared(rec.size) { - s.toHash = append(s.toHash, rec) - - continue - } - - s.sentinels = append(s.sentinels, rec) - s.resolve(rec.path) - } - - for _, rec := range unhashed { - if s.sizeShared(rec.size) { - s.toHash = append(s.toHash, rec) - - continue - } - - delete(s.existing, rec.path) - - s.st.unchanged++ - } - - s.sizes = nil -} - -// sizeShared reports whether at least two census entries have this -// size. Every candidate's own size is in the census exactly once, so -// a second entry means another file (or a record outside the scan -// roots) could share its content. -func (s *scanState) sizeShared(size int64) bool { - i, found := slices.BinarySearch(s.sizes, size) - - return found && i+1 < len(s.sizes) && s.sizes[i+1] == size } // resolve counts one written record as added or updated and marks its @@ -725,44 +625,31 @@ func isDatabaseFile(f fileRec, rules walkRules) bool { }) } -// hashPhase hashes every queued file with hashSignature — the head and -// tail of a file of headTailMin or more, the whole file below that — -// committing completed records to the database in batches as results -// arrive, so a long scan persists its progress as it goes (an -// interrupted scan resumes cheaply: the next run skips everything +// hashPhase hashes every queued file with hashSignature, one read per +// inode run, committing completed records to the database in batches +// as results arrive, so a long scan persists its progress as it goes +// (an interrupted scan resumes cheaply: the next run skips everything // already recorded). A run that fails to hash is warned about and -// skipped; stale records for its paths, if any, are deleted by the -// update phase. -func (s *scanState) hashPhase(ctx context.Context, workers int) error { - runs := hashRuns(s.toHash) - s.toHash = nil - - return s.readRuns(ctx, workers, "hash", runs, hashSignature, s.recordRun) -} - -// readRuns reads runs with the worker pool, one read per inode run, in -// the order given, under the progress display of the phase named label. -// The workers compute each run's hashes with hash, and each result goes -// to record; a run that fails to read is warned about and counted as -// skipped instead. The total counts actual reads, so the display shows -// a real ETA. +// counted as skipped; stale records for its paths, if any, are deleted +// by the update phase. A run whose file no longer exists gets no record +// and no warning, and is not counted as skipped. The total counts actual +// reads, so the display's time remaining is meaningful. // // Returning early — a failed database write, or a cancelled scan — must // not strand the pool: the feeder would park forever on a full jobs // channel and every worker on a full results channel. The deferred stop // is what prevents that. -func (s *scanState) readRuns(ctx context.Context, workers int, - label string, runs [][]fileRec, - hash func(path string, size int64) (string, string, string, error), - record func(ctx context.Context, r hashResult) error, -) error { - pool := startHashPool(ctx, runs, workers, hash) +func (s *scanState) hashPhase(ctx context.Context, workers int) error { + runs := hashRuns(s.toHash) + s.toHash = nil + + pool := startHashPool(ctx, runs, workers, hashSignature) defer pool.stop() // Between results the phase may wait on a file being read or on a // batch of records being written. prog := newProgress(progressParams{ - label: scanPhase(label), unit: scanUnit, total: int64(len(runs)), + label: scanPhase("hash"), unit: scanUnit, total: int64(len(runs)), stalledOn: "disk/db", }) defer prog.finish() @@ -778,15 +665,20 @@ func (s *scanState) readRuns(ctx context.Context, workers int, prog.increment() + // A file removed since the walk is no error: it gets no record. + if errors.Is(r.err, fs.ErrNotExist) { + continue + } + if r.err != nil { s.st.skipped += len(r.run) - prog.warnf("%s %s: %v", label, r.run[0].path, r.err) + prog.warnf("hash %s: %v", r.run[0].path, r.err) continue } - err := record(ctx, r) + err := s.recordRun(ctx, r) if err != nil { return err } @@ -835,10 +727,9 @@ func (s *scanState) commitFullBatch(ctx context.Context) error { } // updatePhase writes the scan's tail under one progress display: the -// final partial batch of hashed records, a hash-less record for every -// size-unique new or changed file, and deletions for every record the -// scan did not verify (vanished files, plus paths that failed to stat -// or hash). +// final partial batch of hashed records, and deletions for every record +// the scan did not verify (vanished files, plus paths that failed to +// stat or hash). func (s *scanState) updatePhase(ctx context.Context) error { deletes := make([]string, 0, len(s.existing)) for path := range s.existing { @@ -850,7 +741,7 @@ func (s *scanState) updatePhase(ctx context.Context) error { s.st.removed = len(deletes) - total := len(s.batch) + len(s.sentinels) + len(deletes) + total := len(s.batch) + len(deletes) prog := newProgress(progressParams{ label: scanPhase("update"), unit: scanUnit, total: int64(total), stalledOn: "db", @@ -865,209 +756,9 @@ func (s *scanState) updatePhase(ctx context.Context) error { s.batch = nil - // Sentinel records are converted in batch-sized chunks rather than - // materialized all at once; a first scan can have millions. - for chunk := range slices.Chunk(s.sentinels, updateBatchSize) { - recs := make([]scanRec, 0, len(chunk)) - for _, rec := range chunk { - recs = append(recs, scanRec{ - size: rec.size, mtime: rec.mtime, path: rec.path, - }) - } - - err = applyBatch(ctx, s.db, recs, nil, prog) - if err != nil { - return err - } - } - return applyChanges(ctx, s.db, nil, deletes, prog) } -// contentPhase fills in the content hash of every record of headTailMin -// or more that lacks one and whose size, head, and tail equal another -// record's, anywhere in the database: records from this scan and -// records stored by earlier scans, inside or outside the roots. Only -// such a file can still be a duplicate, so no other file of headTailMin -// or more is read beyond its end windows. The files are read with the -// hash phase's worker pool and their records written back in batches. A -// failed read is warned about and counted as skipped; the record keeps -// its empty content, so it is never grouped, and a later scan tries -// again. No file on a filesystem in rules that is not persistent is -// read. -func (s *scanState) contentPhase(ctx context.Context, workers int, - rules walkRules, -) error { - toRead, recs, err := s.contentCandidates(ctx, rules) - if err != nil { - return err - } - - err = s.readRuns(ctx, workers, "content", hashRuns(toRead), - hashContentOnly, func(ctx context.Context, r hashResult) error { - // Every path in the run keeps its record's head and tail - // and gains the one content hash read for the run. - for _, f := range r.run { - rec := recs[f.path] - rec.content = r.content - s.batch = append(s.batch, rec) - } - - return s.commitFullBatch(ctx) - }) - if err != nil { - return err - } - - // The last partial batch is written under a display of its own, as - // the update phase writes the hash phase's. - prog := newProgress(progressParams{ - label: scanPhase("content"), unit: scanUnit, - total: int64(len(s.batch)), stalledOn: "db", - }) - defer prog.finish() - - return applyChanges(ctx, s.db, s.batch, nil, prog) -} - -// contentCandidates returns the files the content phase reads, and -// their records by path. Every record contentCandidatesSQL returns has -// its file checked with lstat, whether or not it already has a content -// hash: a file that is gone, is no longer a regular file, has changed -// by the walk's rule, or is on a filesystem in rules that is not -// persistent keeps its record as it is and does not count as a match -// for the others, and any other lstat error is warned about and counted -// as skipped, with the same result. If such a record has no content -// hash, it stays out of duplicate groups; if it has one, it is still -// reported until a scan covering its own tree updates or removes it. -// The files of a group that pass and have no content hash are read only -// if at least minGroupSize of the group's files pass, so a group whose -// other members are all stale costs no reads. Only the records to be -// read are kept. -func (s *scanState) contentCandidates( - ctx context.Context, rules walkRules, -) ([]fileRec, map[string]scanRec, error) { - // The search and the checks take real time on a large database; - // without a display the scan looks hung before the reads begin. - // The display says it is searching until SQLite returns the first - // record. The checks start there, against a count of the records - // the search returns, and so do their rate and ETA. Between records - // they may wait on a file's lstat or on the search. - prog := newProgress(progressParams{ - label: scanPhase("content"), unit: scanUnit, total: -1, - waitingFor: "searching the database", stalledOn: "disk/db", - }) - defer prog.finish() - - total, err := countContentCandidates(ctx, s.db) - if err != nil { - return nil, nil, err - } - - var ( - searching = true // no record has been returned yet - toRead []fileRec - first scanRec // the current group's first record - passed int // the current group's files that passed the check - unread []fileRec // those of them without a content hash - ) - - recs := make(map[string]scanRec) - - // endGroup queues the current group's files to read if at least - // minGroupSize of its files passed, and drops their records if not. - endGroup := func() { - if passed >= minGroupSize { - toRead = append(toRead, unread...) - } else { - for _, f := range unread { - delete(recs, f.path) - } - } - - passed, unread = 0, nil - } - - err = loadContentCandidates(ctx, s.db, func(r scanRec, hashed bool) { - if searching { - searching = false - - prog.setTotal(total) - } - - prog.increment() - - if r.size != first.size || r.head != first.head || r.tail != first.tail { - endGroup() - - first = r - } - - f, ok, err := unchangedFile(r, rules) - if err != nil { - s.st.skipped++ - - prog.warnf("content %s: %v", r.path, err) - } - - if !ok { - return - } - - passed++ - - if !hashed { - unread = append(unread, f) - recs[r.path] = r - } - }) - if err != nil { - return nil, nil, err - } - - // A search that returned no record still ends showing its count. - if searching { - prog.setTotal(total) - } - - endGroup() - - return toRead, recs, nil -} - -// unchangedFile lstats the file r names and returns it for reading if -// it is still the regular file r records: the same size, and an mtime -// no newer than recorded (the walk's change rule). A file that is gone, -// has changed, or is on a filesystem in rules that is not persistent -// reports false; any other lstat error is returned. -func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) { - fi, err := os.Lstat(r.path) - if errors.Is(err, fs.ErrNotExist) { - return fileRec{}, false, nil - } - - if err != nil { - return fileRec{}, false, err - } - - if !fi.Mode().IsRegular() || fi.Size() != r.size || - mtimeAfter(fi.ModTime(), r.mtime) { - return fileRec{}, false, nil - } - - dev, ino := inodeOfInfo(fi) - - // The scan reads nothing on a filesystem that is not persistent, - // whatever record an earlier scan left for the file. - if _, found := rules.nonPersistent[dev]; found { - return fileRec{}, false, nil - } - - return fileRec{ - path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino, - }, true, nil -} - // mtimeAfter reports whether mtime a is later than mtime b. // Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it. func mtimeAfter(a, b time.Time) bool { @@ -1146,10 +837,12 @@ type walkEvent struct { // startWalk seeds every root into the shared walk worker pool and // returns the event stream: one record per regular file, one warning // event per per-path error, and one event per path -x or rules kept the -// walk from entering or recording. The channel is closed when the walk -// completes, and also when ctx is cancelled — every goroutine in the -// pool abandons its blocking send in that case, so the consumer sees a -// truncated but properly terminated stream instead of a stalled one. +// walk from entering or recording. A path that no longer exists when +// the walk reaches it is no error, and gets no event. The channel is +// closed when the walk completes, and also when ctx is cancelled — +// every goroutine in the pool abandons its blocking send in that case, +// so the consumer sees a truncated but properly terminated stream +// instead of a stalled one. func startWalk(ctx context.Context, roots []string, oneFS bool, rules walkRules, workers int, ) <-chan walkEvent { @@ -1182,9 +875,24 @@ func sendEvent(ctx context.Context, events chan<- walkEvent, // operandWarning returns the one-line warning for an operand the walk // does not start from, naming the path and what it is, or "" for one it -// does: a regular file, or a directory not named .zfs. Symlinks are -// never followed, including as operands. +// does: a regular file, or a directory not named .zfs, that is not +// inside a directory named .zfs, neither as given nor with its symlinks +// resolved. Symlinks are never followed, including as operands. func operandWarning(root string, fi fs.FileInfo) string { + // The walk never enters a .zfs directory, so it starts from nothing + // inside one either, not even through a symlink. An operand whose + // symlinks cannot be resolved is checked as given. + resolved, err := filepath.EvalSymlinks(root) + if err != nil { + resolved = root + } + + if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") || + slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") { + return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory", + root) + } + var kind string switch mode := fi.Mode(); { @@ -1216,13 +924,19 @@ func operandWarning(root string, fi fs.FileInfo) string { // one of the scan's own database files in rules, and a directory // operand becomes an initial job. walkableRoots has already dropped // every other operand. One that has changed into something else since -// is warned about and skipped here; it is still a root, so the records -// stored beneath it, other than those under another root, are deleted -// as unverified. +// is warned about and skipped here, and one removed since is skipped +// without a warning. Either is still a root, so the records stored +// beneath it, other than those under another root, are deleted as +// unverified. func seedRoot(ctx context.Context, root string, rules walkRules, events chan<- walkEvent, ) []dirJob { fi, err := os.Lstat(root) + if errors.Is(err, fs.ErrNotExist) { + // Removed since the scan started: no error, and no files. + return nil + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", root, err), @@ -1340,11 +1054,17 @@ func dispatchDirs(ctx context.Context, initial []dirJob, // walkOneDir reads one directory, emitting an event per regular-file // entry and a warning event per unreadable one, and returns the -// subdirectories to descend into. +// subdirectories to descend into. A directory or entry removed since +// the walk found it gets no event. func walkOneDir(ctx context.Context, job dirJob, oneFS bool, rules walkRules, events chan<- walkEvent, ) []dirJob { entries, err := os.ReadDir(job.path) + if errors.Is(err, fs.ErrNotExist) { + // Removed since its parent was read: no error, and no files. + return nil + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", job.path, err), @@ -1387,16 +1107,22 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool, // emitFile stats one regular directory entry and emits its record. // The lstat happens here in the walk worker, while the directory's -// metadata is still hot; a path that fails to stat (or stops being a -// regular file) between the directory read and the lstat is warned -// about and skipped. A file on a filesystem in rules that is not -// persistent, as through a bind mount of one file, is not recorded: it -// gets an event naming it instead. One of the scan's own database files -// in rules gets no event at all. +// metadata is still hot; a path that fails to stat between the +// directory read and the lstat is warned about and skipped, unless it +// no longer exists, and one that has stopped being a regular file is +// skipped. A file on a filesystem in rules that is not persistent, as +// through a bind mount of one file, is not recorded: it gets an event +// naming it instead. One of the scan's own database files in rules gets +// no event at all. func emitFile(ctx context.Context, p string, e fs.DirEntry, rules walkRules, events chan<- walkEvent, ) { info, err := e.Info() + if errors.Is(err, fs.ErrNotExist) { + // Removed since its directory was read: no error. + return + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("stat %s: %v", p, err), @@ -1448,6 +1174,11 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry, } info, err := e.Info() + if errors.Is(err, fs.ErrNotExist) { + // Removed since its parent was read: no error, and no files. + return dirJob{}, false + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", p, err), @@ -1512,13 +1243,12 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) { // hashResult carries the hashes computed for one inode run (or the // error that prevented computing them) from the pool's workers to the -// phase that started the pool: head, tail, and content from -// hashSignature, content alone from hashContentOnly. +// hash phase. Its run leaves out the paths found removed. type hashResult struct { run []fileRec - head string - tail string - content string + head [sha256.Size]byte + tail [sha256.Size]byte + content [sha256.Size]byte err error } @@ -1538,7 +1268,8 @@ type hashPool struct { // hash each run's first path with hash (all paths in a run are hard // links to the same inode) and write one result per run. func startHashPool(ctx context.Context, runs [][]fileRec, workers int, - hash func(path string, size int64) (string, string, string, error), + hash func(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error), ) *hashPool { ctx, cancel := context.WithCancel(ctx) @@ -1603,7 +1334,8 @@ func feedHashJobs(ctx context.Context, runs [][]fileRec, // the hash of only delays that. func hashWorker(ctx context.Context, jobs <-chan []fileRec, results chan<- hashResult, - hash func(path string, size int64) (string, string, string, error), + hash func(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error), ) { for run := range jobs { if ctx.Err() != nil { @@ -1612,6 +1344,13 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec, head, tail, content, err := hash(run[0].path, run[0].size) + // A path removed since the walk is dropped from its run. The + // next path is a hard link to the same file, and is read instead. + for errors.Is(err, fs.ErrNotExist) && len(run) > 1 { + run = run[1:] + head, tail, content, err = hash(run[0].path, run[0].size) + } + select { case results <- hashResult{ run: run, head: head, tail: tail, content: content, err: err, @@ -1654,100 +1393,73 @@ func openRegularFile(path string) (*os.File, error) { return f, nil } -// emptyHash is the lowercase-hex SHA-256 of the empty input: the head, -// tail, and content hash of every zero-length file. -const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" + - "27ae41e4649b934ca495991b7852b855" +// hashSignature computes the three hashes the hash phase records for a +// file: head and tail (see hashEnds) and content (see hashContent). +// With the file size they form its duplicate signature, and two files +// are duplicates only when all four agree. size is the value recorded +// when the file was statted; a zero-length file has constant hashes and +// is never opened. On an error every hash is left zero. +func hashSignature(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error, +) { + var zero [sha256.Size]byte -// hashSignature computes the hashes the hash phase records for a file -// whose size is shared; with the file size they form its duplicate -// signature. A file below headTailMin is hashed in full and its -// whole-file SHA-256 is returned as head, tail, and content alike — -// that range takes no separate end-window step. For a file at or above -// headTailMin only the head and tail are computed, the SHA-256 of its -// first and last headTailWindow bytes, and content is returned empty: -// the content phase computes it with hashContentOnly once the file's -// size, head, and tail match another file's. Two files are duplicates -// only when all four agree; any mismatch means not a duplicate. size -// is the value recorded when the file was statted; a zero-length file -// has constant hashes and is never opened. -func hashSignature(path string, size int64) (string, string, string, error) { if size == 0 { - return emptyHash, emptyHash, emptyHash, nil + empty := sha256.Sum256(nil) + + return empty, empty, empty, nil } f, err := openRegularFile(path) if err != nil { - return "", "", "", err + return zero, zero, zero, err } defer func() { _ = f.Close() }() - // Below the threshold the whole file is hashed directly, with no - // end-window step: head and tail both carry the whole-file hash. - if size < int64(headTailMin) { - content, err := hashWhole(f, size) - if err != nil { - return "", "", "", err - } - - return content, content, content, nil - } - head, tail, err := hashEnds(f, size) if err != nil { - return "", "", "", err + return zero, zero, zero, err } - return head, tail, "", nil -} - -// hashContentOnly returns the content hash of the file at path, which -// is at least headTailMin bytes: the content phase's read. head and -// tail are returned empty, because the content phase keeps the ones its -// records already hold. -func hashContentOnly(path string, size int64) (string, string, string, error) { - f, err := openRegularFile(path) - if err != nil { - return "", "", "", err - } - - defer func() { _ = f.Close() }() - content, err := hashContent(f, size) + if err != nil { + return zero, zero, zero, err + } - return "", "", content, err + return head, tail, content, nil } // hashEnds returns the SHA-256 of the first and last headTailWindow -// bytes of f. It is called only for files at least headTailMin, which -// is far larger than two windows, so the windows never overlap and both -// reads are always full. -func hashEnds(f *os.File, size int64) (string, string, error) { - buf := make([]byte, headTailWindow) +// bytes of f. In a file no larger than headTailWindow, each window is +// the whole file. Reading fewer bytes than asked means the file shrank +// between the stat and the hash, and is an error. +func hashEnds(f *os.File, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, error, +) { + var zero [sha256.Size]byte + + buf := make([]byte, min(int64(headTailWindow), size)) _, err := f.ReadAt(buf, 0) if err != nil { - return "", "", err + return zero, zero, err } - h := sha256.Sum256(buf) - head := hex.EncodeToString(h[:]) + head := sha256.Sum256(buf) - _, err = f.ReadAt(buf, size-int64(headTailWindow)) + _, err = f.ReadAt(buf, size-int64(len(buf))) if err != nil { - return "", "", err + return zero, zero, err } - t := sha256.Sum256(buf) - - return head, hex.EncodeToString(t[:]), nil + return head, sha256.Sum256(buf), nil } -// hashContent returns the content-rung hash of f: the SHA-256 of the -// whole file when it is smaller than wholeFileMax, or of sampled -// windows when it is that size or larger. -func hashContent(f *os.File, size int64) (string, error) { +// hashContent returns the content hash of f: the SHA-256 of the whole +// file when it is smaller than wholeFileMax, or of sampled windows when +// it is that size or larger. +func hashContent(f *os.File, size int64) ([sha256.Size]byte, error) { if size >= int64(wholeFileMax) { return hashSamples(f, size) } @@ -1760,30 +1472,30 @@ func hashContent(f *os.File, size int64) (string, error) { // reads. Reading fewer than size bytes means the file shrank between // the stat and the hash; that is an error rather than a hash of content // that no longer matches the recorded size. -func hashWhole(f *os.File, size int64) (string, error) { +func hashWhole(f *os.File, size int64) ([sha256.Size]byte, error) { h := sha256.New() n, err := io.Copy(h, io.NewSectionReader(f, 0, size)) if err != nil { - return "", err + return [sha256.Size]byte{}, err } if n != size { - return "", fmt.Errorf("read %d of %d bytes: %w", n, size, - io.ErrUnexpectedEOF) + return [sha256.Size]byte{}, fmt.Errorf("read %d of %d bytes: %w", + n, size, io.ErrUnexpectedEOF) } - return hex.EncodeToString(h.Sum(nil)), nil + return [sha256.Size]byte(h.Sum(nil)), nil } // hashSamples feeds sampleWindow bytes at each gigabyte-aligned offset // (0, sampleStride, 2*sampleStride, ... while inside the file), in -// order, into one hash, each window truncated at end of file. This is -// the probabilistic large-file rung: two files of equal size agreeing -// on every sample are reported as duplicates without every byte being -// read. Because size is part of the signature, files of different sizes -// never reach this comparison, so the sample boundaries always align. -func hashSamples(f *os.File, size int64) (string, error) { +// order, into one hash, each window truncated at end of file. Two files +// of equal size agreeing on every sample are reported as duplicates +// without every byte being read. Because size is part of the signature, +// only files of one size are compared, so their sample boundaries +// always align. +func hashSamples(f *os.File, size int64) ([sha256.Size]byte, error) { h := sha256.New() buf := make([]byte, sampleWindow) @@ -1792,11 +1504,11 @@ func hashSamples(f *os.File, size int64) (string, error) { _, err := f.ReadAt(buf[:n], off) if err != nil { - return "", err + return [sha256.Size]byte{}, err } h.Write(buf[:n]) } - return hex.EncodeToString(h.Sum(nil)), nil + return [sha256.Size]byte(h.Sum(nil)), nil } diff --git a/scan_test.go b/scan_test.go index 1d719ef..58ab1db 100644 --- a/scan_test.go +++ b/scan_test.go @@ -5,7 +5,6 @@ import ( "context" "crypto/sha256" "database/sql" - "encoding/hex" "errors" "fmt" "io" @@ -43,13 +42,6 @@ func writeFile(t *testing.T, dir, name string, data []byte) string { return p } -// hexSum returns the lowercase-hex SHA-256 of data. -func hexSum(data []byte) string { - s := sha256.Sum256(data) - - return hex.EncodeToString(s[:]) -} - // pattern returns n bytes of deterministic content seeded by tag. func pattern(tag byte, n int) []byte { data := make([]byte, n) @@ -62,7 +54,9 @@ func pattern(tag byte, n int) []byte { // sig returns a file's full signature (head, tail, content), failing the // test on any error. -func sig(t *testing.T, path string, size int64) (string, string, string) { +func sig(t *testing.T, path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, +) { t.Helper() head, tail, content, err := hashSignature(path, size) @@ -73,10 +67,12 @@ func sig(t *testing.T, path string, size int64) (string, string, string) { return head, tail, content } -// TestHashSignatureBelowThreshold verifies that a file below headTailMin -// is hashed in full and compared directly: head, tail, and content all -// carry the whole-file SHA-256, with no separate end-window step. -func TestHashSignatureBelowThreshold(t *testing.T) { +// TestHashSignature checks the three hashes of a file under +// wholeFileMax: head and tail are the SHA-256 of its first and last +// headTailWindow bytes, and content that of the whole file. In a file +// no larger than headTailWindow each window is the whole file, so all +// three are the whole-file hash. +func TestHashSignature(t *testing.T) { t.Parallel() dir := t.TempDir() @@ -87,8 +83,7 @@ func TestHashSignatureBelowThreshold(t *testing.T) { }{ {"one-byte", []byte("x")}, {"one-window", pattern(1, headTailWindow)}, - {"several-windows", pattern(2, 3*headTailWindow)}, - {"near-threshold", pattern(3, headTailMin-1)}, + {"several-windows", pattern(2, 3*headTailWindow+1)}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { @@ -98,31 +93,31 @@ func TestHashSignatureBelowThreshold(t *testing.T) { head, tail, content := sig(t, p, int64(len(c.data))) - whole := hexSum(c.data) - if head != whole || tail != whole || content != whole { - t.Errorf("head=%s tail=%s content=%s, want all whole-file %s", - head, tail, content, whole) + window := min(len(c.data), headTailWindow) + wantHead := sha256.Sum256(c.data[:window]) + wantTail := sha256.Sum256(c.data[len(c.data)-window:]) + wantContent := sha256.Sum256(c.data) + + if head != wantHead || tail != wantTail || content != wantContent { + t.Errorf("head=%x tail=%x content=%x, want %x %x %x", + head, tail, content, wantHead, wantTail, wantContent) } }) } } -// TestHashSignatureEnds exercises the head and tail rungs, which apply -// only to files at least headTailMin. Sparse files keep the fixtures -// cheap: a difference in the first window changes only head, a -// difference in the last window changes only tail, and a difference -// between the windows changes neither end hash but does change the -// whole-file content rung (the file is below wholeFileMax). -// hashSignature leaves the content hash of a file this size to the -// content phase, so that rung is checked through a scan. +// TestHashSignatureEnds checks which hashes a one-byte difference +// changes in a file under wholeFileMax. Sparse files keep the fixtures +// cheap: a difference in the first window changes head but not tail, +// one in the last window changes tail but not head, and one between the +// windows changes neither but does change the whole-file content hash. func TestHashSignatureEnds(t *testing.T) { t.Parallel() dir := t.TempDir() - // Between headTailMin and wholeFileMax: the end-window gate is active - // and the content rung is a whole-file hash. - const size = int64(headTailMin + 2*1024*1024) + // Room for a byte between the two windows. + const size = int64(4 * headTailWindow) base := sparseFile(t, dir, "ends-base", size) headDiff := sparseFile(t, dir, "ends-head", size) @@ -134,9 +129,6 @@ func TestHashSignatureEnds(t *testing.T) { pokeAt(t, midDiff, size/2, []byte{1}) bHead, bTail, bContent := sig(t, base, size) - if bContent != "" { - t.Errorf("content = %q, want none from the hash phase", bContent) - } h, tl, _ := sig(t, headDiff, size) if h == bHead { @@ -156,16 +148,14 @@ func TestHashSignatureEnds(t *testing.T) { t.Error("a byte in the last window changed head") } - h, tl, _ = sig(t, midDiff, size) + h, tl, c := sig(t, midDiff, size) if h != bHead || tl != bTail { t.Error("a byte between the windows changed an end hash") } - // base and midDiff match on size, head, and tail, so the scan reads - // both for their content hashes. - c := scanContents(t, dir, base, midDiff) - if c[midDiff] == c[base] { - t.Error("whole-file content rung ignored a byte between the windows") + if c == bContent { + t.Error("the whole-file content hash ignored a byte between the " + + "windows") } } @@ -174,22 +164,25 @@ func TestHashSignatureErrors(t *testing.T) { dir := t.TempDir() - // A missing file: an error, and every hash left empty. + var zero [sha256.Size]byte + + // A missing file: an error, and every hash left zero. head, tail, content, err := hashSignature(filepath.Join(dir, "missing"), 1) if err == nil { t.Error("no error for a missing file") } - if head != "" || tail != "" || content != "" { - t.Errorf("missing file returned hashes: %q %q %q", head, tail, content) + if head != zero || tail != zero || content != zero { + t.Errorf("missing file returned hashes: %x %x %x", head, tail, content) } // A zero-length file has constant hashes and is never opened: even // a missing path succeeds. + empty := sha256.Sum256(nil) + head, tail, content, err = hashSignature(filepath.Join(dir, "missing"), 0) - if err != nil || - head != emptyHash || tail != emptyHash || content != emptyHash { - t.Errorf("empty: head=%q tail=%q content=%q err=%v, "+ + if err != nil || head != empty || tail != empty || content != empty { + t.Errorf("empty: head=%x tail=%x content=%x err=%v, "+ "want constant hashes", head, tail, content, err) } @@ -202,8 +195,8 @@ func TestHashSignatureErrors(t *testing.T) { t.Error("no error when the stat size exceeds the file size") } - if head != "" || tail != "" || content != "" { - t.Errorf("shrunk file returned hashes: %q %q %q", head, tail, content) + if head != zero || tail != zero || content != zero { + t.Errorf("shrunk file returned hashes: %x %x %x", head, tail, content) } } @@ -253,79 +246,46 @@ func pokeAt(t *testing.T, path string, off int64, data []byte) { } } -// scanContents scans dir into a fresh database and returns the content -// hash recorded for each file, by path, failing the test if one of want -// has none. A file of headTailMin or more gets a content hash only when -// it is scanned with a file of the same size, head, and tail. -func scanContents(t *testing.T, dir string, - want ...string, -) map[string]string { - t.Helper() - - db := openTestDB(t) - syncTree(t, db, dir) - - contents := make(map[string]string) - for _, r := range dbRecords(t, db) { - contents[r.path] = r.content - } - - for _, p := range want { - if contents[p] == "" { - t.Fatalf("%s: no content hash", p) - } - } - - return contents -} - -// TestContentRungBoundary checks the 50 MiB boundary between the two -// content rungs: just below it the whole file is hashed and any byte -// difference shows; at the boundary only the gigabyte-spaced samples are -// hashed, so a difference outside a sample window is invisible. The -// files of each pair match on size, head, and tail, so the scan reads -// both for their content hashes. -func TestContentRungBoundary(t *testing.T) { +// TestContentHashBoundary checks the 100 MiB boundary between the two +// content hashes through a scan and report. The files of each pair +// differ in one byte that neither end window nor any sample covers. +// Just below the boundary the whole file is hashed, so that pair is not +// reported. At the boundary only the samples are hashed, so that pair +// is reported as duplicates. +func TestContentHashBoundary(t *testing.T) { t.Parallel() dir := t.TempDir() + db := openTestDB(t) // A byte that lands outside the single [0, sampleWindow) sample a - // sub-gigabyte file has, but well inside the file. + // sub-gigabyte file has, and outside both end windows. const off = 10 * 1024 * 1024 - // Just under the boundary: the whole-file rung sees the poked byte. - under := int64(wholeFileMax - 1) - underBase := sparseFile(t, dir, "under-base", under) - underPoked := sparseFile(t, dir, "under-poked", under) + under := sparseFile(t, dir, "under-a", wholeFileMax-1) + underPoked := sparseFile(t, dir, "under-b", wholeFileMax-1) + at := sparseFile(t, dir, "at-a", wholeFileMax) + atPoked := sparseFile(t, dir, "at-b", wholeFileMax) pokeAt(t, underPoked, off, []byte{1}) - - // At the boundary: only [0, sampleWindow) is sampled, so the poked - // byte at off is invisible and the two content hashes match. - at := int64(wholeFileMax) - atBase := sparseFile(t, dir, "at-base", at) - atPoked := sparseFile(t, dir, "at-poked", at) - pokeAt(t, atPoked, off, []byte{1}) - c := scanContents(t, dir, underBase, underPoked, atBase, atPoked) + syncTree(t, db, dir) - if c[underBase] == c[underPoked] { - t.Error("whole-file rung ignored a byte difference below wholeFileMax") - } + want := []string{at, atPoked} - if c[atBase] != c[atPoked] { - t.Error("sampled rung saw a byte outside every sample window") + groups := dupeGroups(t, db) + if len(groups) != 1 || !slices.Equal(groups[0].paths, want) { + t.Fatalf("groups = %+v, want only the pair %q, not %q", groups, want, + []string{under, underPoked}) } } -// TestContentRungMultiGigabyte exercises the sampled rung across several -// gigabytes using sparse files: a difference inside the third sample -// window (at offset 2*sampleStride) changes the hash, while a difference -// in the gap after it does not. The three files match on size, head, -// and tail, so the scan reads each for its content hash. -func TestContentRungMultiGigabyte(t *testing.T) { +// TestContentHashMultiGigabyte exercises the sampled content hash +// across several gigabytes using sparse files: a difference inside the +// third sample window (at offset 2*sampleStride) changes the hash, +// while a difference in the gap after it does not. +func TestContentHashMultiGigabyte(t *testing.T) { t.Parallel() dir := t.TempDir() @@ -343,433 +303,19 @@ func TestContentRungMultiGigabyte(t *testing.T) { pokeAt(t, inSample, thirdSample, []byte{1}) pokeAt(t, inGap, gap, []byte{1}) - c := scanContents(t, dir, base, inSample, inGap) + _, _, cBase := sig(t, base, size) + _, _, cInSample := sig(t, inSample, size) + _, _, cInGap := sig(t, inGap, size) - if c[inSample] == c[base] { + if cInSample == cBase { t.Error("sample at 2 GiB was not read: difference there was invisible") } - if c[inGap] != c[base] { + if cInGap != cBase { t.Error("a byte in an unsampled gap changed the content hash") } } -// sparseFileWithoutMatch writes name in dir as a sparse file of size -// bytes, next to another file of that size whose first byte differs. A -// scan then reads the file's head and tail, since its size is shared, -// but finds no file matching them, so it gets no content hash. -func sparseFileWithoutMatch(t *testing.T, dir, name string, - size int64, -) string { - t.Helper() - - p := sparseFile(t, dir, name, size) - other := sparseFile(t, dir, name+"-other-head", size) - - pokeAt(t, other, 0, []byte{1}) - - return p -} - -// TestScanContentGate checks that a file of headTailMin or more is read -// for its content hash only when its size, head, and tail match another -// file's: a same-size pair whose heads differ and one whose tails differ -// get no content hash and are not reported, while an identical pair is -// read and reported. -func TestScanContentGate(t *testing.T) { - t.Parallel() - - dir := t.TempDir() - db := openTestDB(t) - - // Three sizes, so that no pair meets another. - headA := sparseFile(t, dir, "head-a", headTailMin) - headB := sparseFile(t, dir, "head-b", headTailMin) - tailA := sparseFile(t, dir, "tail-a", headTailMin+1) - tailB := sparseFile(t, dir, "tail-b", headTailMin+1) - same := []string{ - sparseFile(t, dir, "same-a", headTailMin+2), - sparseFile(t, dir, "same-b", headTailMin+2), - } - - pokeAt(t, headB, 0, []byte{1}) - pokeAt(t, tailB, headTailMin, []byte{1}) // its last byte - - syncTree(t, db, dir) - - recs := dbRecords(t, db) - for _, p := range []string{headA, headB, tailA, tailB} { - r := recordByPath(t, recs, p) - if r.head == "" || r.tail == "" || r.content != "" { - t.Errorf("%s: head = %q tail = %q content = %q, "+ - "want head and tail only", p, r.head, r.tail, r.content) - } - } - - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, same) { - t.Fatalf("groups = %+v, want only the identical pair %q", - groups, same) - } -} - -// TestScanContentAcrossOperands checks that a stored file gets its -// content hash when a later scan of a separate operand brings its -// match: tree A's file has a head and tail but no content hash until -// tree B, holding an identical file, is scanned. -func TestScanContentAcrossOperands(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - a := sparseFileWithoutMatch(t, t.TempDir(), "a", headTailMin) - - syncTree(t, db, filepath.Dir(a)) - - if r := recordByPath(t, dbRecords(t, db), a); r.head == "" || r.content != "" { - t.Fatalf("after scanning A: %+v, want head and tail only", r) - } - - b := sparseFile(t, t.TempDir(), "b", headTailMin) - - syncTree(t, db, filepath.Dir(b)) - - recs := dbRecords(t, db) - if r := recordByPath(t, recs, a); r.content == "" { - t.Fatalf("after scanning B: %+v, want A's file content-hashed", r) - } - - want := []string{a, b} - slices.Sort(want) - - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, want) { - t.Fatalf("groups = %+v, want the pair %q", groups, want) - } -} - -// TestScanContentWithinOperand checks that a rescan adding a match next -// to an unchanged stored file gives the stored file its content hash, -// though the hash phase leaves it alone as unchanged. -func TestScanContentWithinOperand(t *testing.T) { - t.Parallel() - - dir := t.TempDir() - db := openTestDB(t) - stored := sparseFileWithoutMatch(t, dir, "d1", headTailMin) - - syncTree(t, db, dir) - - added := sparseFile(t, dir, "d2", headTailMin) - - st := syncTree(t, db, dir) - if st != (scanStats{walked: 3, added: 1, unchanged: 2}) { - t.Fatalf("rescan stats = %+v, want 1 added 2 unchanged", st) - } - - want := []string{stored, added} - - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, want) { - t.Fatalf("groups = %+v, want the pair %q", groups, want) - } -} - -// TestScanContentStalePartners checks that a stored file outside the -// operand that has vanished, or changed, since it was recorded is not -// read, and that its match inside the operand is not read either: the -// match has no other partner left, so neither gets a content hash and -// no duplicate is reported. -func TestScanContentStalePartners(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - dirA := t.TempDir() - gone := sparseFileWithoutMatch(t, dirA, "gone", headTailMin) - changed := sparseFileWithoutMatch(t, dirA, "changed", headTailMin+1) - - syncTree(t, db, dirA) - - before := dbRecords(t, db) - - err := os.Remove(gone) - if err != nil { - t.Fatal(err) - } - - future := time.Now().Add(time.Hour) - - err = os.Chtimes(changed, future, future) - if err != nil { - t.Fatal(err) - } - - dirB := t.TempDir() - sparseFile(t, dirB, "gone-copy", headTailMin) - sparseFile(t, dirB, "changed-copy", headTailMin+1) - - st := syncTree(t, db, dirB) - if st != (scanStats{walked: 2, added: 2}) { - t.Errorf("stats = %+v, want 2 added and nothing skipped", st) - } - - recs := dbRecords(t, db) - for _, r := range recs { - if r.content != "" { - t.Errorf("%s: content = %q, want none: its only match is stale", - r.path, r.content) - } - } - - for _, old := range before { - if r := recordByPath(t, recs, old.path); r != old { - t.Errorf("record = %+v, want it left as %+v", r, old) - } - } - - if groups := dupeGroups(t, db); len(groups) != 0 { - t.Errorf("groups = %+v, want none", groups) - } -} - -// TestScanContentHashedStalePartners checks that stored matches outside -// the operand that already have a content hash are checked like any -// other: once one has vanished and the other has changed, a copy of -// them scanned in another tree has no match left, so it is not read and -// is not reported as their duplicate. -func TestScanContentHashedStalePartners(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - dirA := t.TempDir() - stored := []string{ - sparseFile(t, dirA, "changed", headTailMin), - sparseFile(t, dirA, "gone", headTailMin), - } - - // The two stored files match, so this scan gives both a content - // hash. - syncTree(t, db, dirA) - - err := os.Remove(stored[1]) - if err != nil { - t.Fatal(err) - } - - future := time.Now().Add(time.Hour) - - err = os.Chtimes(stored[0], future, future) - if err != nil { - t.Fatal(err) - } - - b := sparseFile(t, t.TempDir(), "copy", headTailMin) - - st := syncTree(t, db, filepath.Dir(b)) - if st != (scanStats{walked: 1, added: 1}) { - t.Errorf("stats = %+v, want 1 added and nothing skipped", st) - } - - recs := dbRecords(t, db) - if r := recordByPath(t, recs, b); r.content != "" { - t.Errorf("copy: content = %q, want none: its only matches are stale", - r.content) - } - - // The stored records lie outside the operand and are left as they - // are, so they still group with each other, but not with the copy. - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, stored) { - t.Errorf("groups = %+v, want only the stored pair %q", groups, stored) - } -} - -// TestScanContentSameSecondRewrite is TestScanContentStalePartners for -// a stored file rewritten in place at the same size with an mtime later -// in the same second than recorded: the file counts as changed, so -// neither it nor its match inside the operand is read. -func TestScanContentSameSecondRewrite(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - dirA := t.TempDir() - changed := sparseFileWithoutMatch(t, dirA, "changed", headTailMin) - - first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC) - - err := os.Chtimes(changed, first, first) - if err != nil { - t.Fatal(err) - } - - syncTree(t, db, dirA) - - before := dbRecords(t, db) - - // Rewrite one byte in place, keeping the size. - pokeAt(t, changed, headTailMin/2, []byte{1}) - - later := first.Add(500 * time.Millisecond) - - err = os.Chtimes(changed, later, later) - if err != nil { - t.Fatal(err) - } - - dirB := t.TempDir() - sparseFile(t, dirB, "changed-copy", headTailMin) - - st := syncTree(t, db, dirB) - if st != (scanStats{walked: 1, added: 1}) { - t.Errorf("stats = %+v, want 1 added and nothing skipped", st) - } - - recs := dbRecords(t, db) - for _, r := range recs { - if r.content != "" { - t.Errorf("%s: content = %q, want none: its only match is stale", - r.path, r.content) - } - } - - for _, old := range before { - if r := recordByPath(t, recs, old.path); r != old { - t.Errorf("record = %+v, want it left as %+v", r, old) - } - } - - if groups := dupeGroups(t, db); len(groups) != 0 { - t.Errorf("groups = %+v, want none", groups) - } -} - -// TestScanContentReadFailure checks that a failed content read is -// counted as skipped and leaves the record without a content hash, and -// that a later scan tries the read again. -func TestScanContentReadFailure(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - a := sparseFileWithoutMatch(t, t.TempDir(), "a", headTailMin) - - syncTree(t, db, filepath.Dir(a)) - - // lstat still works on the unreadable file, so it passes the check - // and fails only when it is read. - err := os.Chmod(a, 0) - if err != nil { - t.Fatal(err) - } - - dirB := t.TempDir() - b := sparseFile(t, dirB, "b", headTailMin) - - st := syncTree(t, db, dirB) - if st != (scanStats{walked: 1, added: 1, skipped: 1}) { - t.Fatalf("stats = %+v, want 1 added 1 skipped", st) - } - - if r := recordByPath(t, dbRecords(t, db), a); r.content != "" { - t.Fatalf("unreadable file: %+v, want no content hash", r) - } - - err = os.Chmod(a, 0o600) - if err != nil { - t.Fatal(err) - } - - st = syncTree(t, db, dirB) - if st != (scanStats{walked: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 unchanged", st) - } - - want := []string{a, b} - slices.Sort(want) - - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, want) { - t.Fatalf("groups = %+v, want the pair %q after the retry", - groups, want) - } -} - -// TestScanContentCheckError checks that a stored file the content phase -// cannot lstat, for a reason other than its being gone, is counted as -// skipped and does not count as a match. -func TestScanContentCheckError(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - sub := filepath.Join(t.TempDir(), "sub") - - err := os.Mkdir(sub, 0o700) - if err != nil { - t.Fatal(err) - } - - sparseFileWithoutMatch(t, sub, "a", headTailMin) - syncTree(t, db, sub) - - // Without search permission on its directory, the stored file's - // lstat fails with permission denied. - err = os.Chmod(sub, 0) - if err != nil { - t.Fatal(err) - } - - t.Cleanup(func() { - //nolint:gosec // removing the directory needs its search bit back - _ = os.Chmod(sub, 0o700) - }) - - b := sparseFile(t, t.TempDir(), "b", headTailMin) - - st := syncTree(t, db, filepath.Dir(b)) - if st != (scanStats{walked: 1, added: 1, skipped: 1}) { - t.Fatalf("stats = %+v, want 1 added 1 skipped", st) - } - - if r := recordByPath(t, dbRecords(t, db), b); r.content != "" { - t.Errorf("b: content = %q, want none: its only match could not be "+ - "checked", r.content) - } -} - -// TestScanContentHardlinks checks that the content phase stores the -// content hash of a hard-linked file on every one of its links. -func TestScanContentHardlinks(t *testing.T) { - t.Parallel() - - dir := t.TempDir() - db := openTestDB(t) - a := sparseFile(t, dir, "a", headTailMin) - b := filepath.Join(dir, "b") - - err := os.Link(a, b) - if err != nil { - t.Fatal(err) - } - - c := sparseFile(t, dir, "copy", headTailMin) - - st := syncTree(t, db, dir) - if st != (scanStats{walked: 3, added: 3}) { - t.Fatalf("stats = %+v, want 3 added", st) - } - - recs := dbRecords(t, db) - - want := recordByPath(t, recs, c).content - if want == "" { - t.Fatal("the copy has no content hash") - } - - for _, p := range []string{a, b} { - if got := recordByPath(t, recs, p).content; got != want { - t.Errorf("%s: content = %q, want %q", p, got, want) - } - } -} - // collectWalk runs a walk over roots and returns the emitted records // and the number of warning events. func collectWalk(t *testing.T, roots []string, oneFS bool, @@ -1137,12 +683,114 @@ func TestSubdirJobStatError(t *testing.T) { } } +// TestWalkPassesOverRemovedPaths removes a file and a directory after +// their parent was read, as when they are deleted while a scan runs. +// The walk then reaches them, the directory also as one to read and as +// an operand. None of that is an error: the walk sends no event. +func TestWalkPassesOverRemovedPaths(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + file := writeFile(t, dir, "file", []byte("x")) + writeFile(t, dir, "sub/f", []byte("y")) + sub := filepath.Join(dir, "sub") + fileEntry := dirEntryFor(t, dir, "file") + subEntry := dirEntryFor(t, dir, "sub") + + for _, p := range []string{file, sub} { + err := os.RemoveAll(p) + if err != nil { + t.Fatal(err) + } + } + + ctx := t.Context() + events := make(chan walkEvent, 4) + + emitFile(ctx, file, fileEntry, walkRules{}, events) + + _, descend := subdirJob(ctx, sub, subEntry, dirJob{path: dir}, false, + walkRules{}, events) + subs := walkOneDir(ctx, dirJob{path: sub}, false, walkRules{}, events) + jobs := seedRoot(ctx, sub, walkRules{}, events) + + close(events) + + for ev := range events { + t.Errorf("event %+v for a removed path, want none", ev) + } + + if descend || len(subs) != 0 || len(jobs) != 0 { + t.Errorf("descend = %t, subdirectories %+v, jobs %+v, want none", + descend, subs, jobs) + } +} + +// TestScanWarnsAboutUnreadablePaths checks that a permission error is +// still warned about and counted as skipped, unlike a removed path. The +// walk reaches an unreadable directory, and a file in a directory that +// can be read but not searched. Another such file is an operand. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestScanWarnsAboutUnreadablePaths(t *testing.T) { + if os.Geteuid() == 0 { + t.Skip("root ignores file permissions") + } + + stderr := captureStderr(t) + dir := t.TempDir() + tree := filepath.Join(dir, "tree") + walked := writeFile(t, tree, "unsearchable/f", []byte("x")) + operand := writeFile(t, dir, "unsearchable/f", []byte("y")) + unreadable := filepath.Join(tree, "unreadable") + + err := os.Mkdir(unreadable, 0o750) + if err != nil { + t.Fatal(err) + } + + modes := map[string]fs.FileMode{ + unreadable: 0, + filepath.Dir(walked): 0o400, + filepath.Dir(operand): 0o400, + } + + for p, mode := range modes { + err = os.Chmod(p, mode) + if err != nil { + t.Fatal(err) + } + + // Runs before t.TempDir's own cleanup, which must delete the files. + t.Cleanup(func() { + //nolint:gosec // removing the files needs the search bit back + _ = os.Chmod(p, 0o700) + }) + } + + st := syncTree(t, openTestDB(t), tree, operand) + if st != (scanStats{skipped: 3}) { + t.Errorf("stats = %+v, want 3 skipped and nothing else", st) + } + + got := stderr() + for _, warning := range []string{ + "walk " + unreadable + ": open " + unreadable + ": permission denied\n", + "stat " + walked + ": lstat " + walked + ": permission denied\n", + "walk " + operand + ": lstat " + operand + ": permission denied\n", + } { + if !strings.Contains(got, warning) { + t.Errorf("stderr = %q, want %q", got, warning) + } + } +} + // syncOneFilesystem scans roots into db with -x and returns the scan // stats. Every device pruneRoots looks up, and the device each // operand's walk compares its subdirectories with, comes from devices // instead of from lstat, so a test can put a path on a filesystem that -// no directory here is on. Unlike runPhases, it skips walkableRoots, -// the check for a cancelled walk, and the content phase. +// no directory here is on. Unlike runPhases, it skips walkableRoots and +// the check for a cancelled walk. func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string, devices map[string]uint64, ) scanStats { @@ -1179,8 +827,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string, dispatchDirs(ctx, initial, jobs, subdirs) - changed, unhashed := s.walkPhase(events, roots) - s.partition(changed, unhashed) + s.walkPhase(events, roots) err = s.hashPhase(ctx, 2) if err != nil { @@ -1198,11 +845,10 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string, // TestScanOneFilesystemKeepsOtherFilesystemRecords checks that with -x a // record below a directory the walk does not enter, because it is on // another filesystem than its operand, counts as outside the operands: -// it is not deleted, and its size joins the size census. The record of -// a gone file that is not below that directory is still deleted, even -// when its path begins with the directory's name. The operand is given -// a device that no directory here is on, so no second real filesystem -// is needed. +// it is not deleted. The record of a gone file that is not below that +// directory is still deleted, even when its path begins with the +// directory's name. The operand is given a device that no directory +// here is on, so no second real filesystem is needed. func TestScanOneFilesystemKeepsOtherFilesystemRecords(t *testing.T) { t.Parallel() @@ -1220,10 +866,6 @@ func TestScanOneFilesystemKeepsOtherFilesystemRecords(t *testing.T) { t.Fatal(err) } - // The same size as mnt/f, so it is hashed only if mnt/f's size is - // in the census. - added := writeFile(t, dir, "added", pattern(2, 100)) - info, err := os.Lstat(dir) if err != nil { t.Fatal(err) @@ -1239,11 +881,6 @@ func TestScanOneFilesystemKeepsOtherFilesystemRecords(t *testing.T) { recs := dbRecords(t, db) recordByPath(t, recs, mounted) - if recordByPath(t, recs, added).head == "" { - t.Errorf("%s not hashed: the census lacks the size of %s", - added, mounted) - } - if slices.Contains(recordPaths(recs), gone) { t.Errorf("record for %s kept: the file is gone and not below mnt", gone) @@ -1591,10 +1228,9 @@ func TestScanPipeline(t *testing.T) { // TestScanProgressNamesEachPhase checks the plain progress lines of a // first scan, and of a rescan of one operand holding two of the // database's records. Each phase's first line names it with its place -// among the five: load counts the records before reading them, the -// content phase searches the database, and the walk estimates its total -// from the records under its operand, or says its ETA is unknown when -// there are none. +// among the four: load counts the records before reading them, and the +// walk estimates its total from the records under its operand, or says +// its time remaining is unknown when there are none. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestScanProgressNamesEachPhase(t *testing.T) { @@ -1609,16 +1245,15 @@ func TestScanProgressNamesEachPhase(t *testing.T) { syncTree(t, db, filepath.Join(dir, "a")) rescan := stderr()[len(first):] - load := "[1/5] load: counting the records, elapsed 0s, eta unknown" - content := "[5/5] content: searching the database, elapsed 0s, eta unknown" + load := "[1/4] load: counting the records, elapsed 0s, remain unknown" cases := map[string][]string{ first: { - load, "[2/5] walk: 0 files, 0 files/s, elapsed 0s, eta unknown", - "[3/5] hash: [0/", "[4/5] update: [0/", content, + load, "[2/4] walk: 0 files, 0 files/s, elapsed 0s, remain unknown", + "[3/4] hash: [0/", "[4/4] update: [0/", }, rescan: { - load, "[2/5] walk: [0/~2] 0%, 0 files/s, elapsed 0s, eta unknown", - "[3/5] hash: [0/", "[4/5] update: [0/", content, + load, "[2/4] walk: [0/~2] 0%, 0 files/s, elapsed 0s, remain unknown", + "[3/4] hash: [0/", "[4/4] update: [0/", }, } @@ -1644,7 +1279,7 @@ func TestScanProgressNamesEachPhase(t *testing.T) { } // The load phase's total is the count of the records. - loaded := fmt.Sprintf("\n[1/5] load: [%d/%d] 100%%, ", + loaded := fmt.Sprintf("\n[1/4] load: [%d/%d] 100%%, ", smokeTreeFiles, smokeTreeFiles) if !strings.Contains(rescan, loaded) { t.Errorf("rescan stderr = %q, want %q", rescan, loaded) @@ -1657,8 +1292,7 @@ func TestSyncScanUnchangedReuse(t *testing.T) { dir := t.TempDir() db := openTestDB(t) a := writeFile(t, dir, "a.bin", pattern(1, 500)) - - writeFile(t, dir, "b.bin", pattern(2, 600)) + b := writeFile(t, dir, "b.bin", pattern(2, 600)) st := syncTree(t, db, dir) if st != (scanStats{walked: 2, added: 2}) { @@ -1666,22 +1300,33 @@ func TestSyncScanUnchangedReuse(t *testing.T) { } // An immediate rescan reuses every record without reading file - // contents. Prove the files are not re-read by corrupting a stored - // hash and observing that it survives the rescan. + // contents. Prove it: make every file unreadable, so that a read + // would fail and count as skipped, and corrupt a stored hash, which + // must survive the rescan. + for _, p := range []string{a, b} { + err := os.Chmod(p, 0) + if err != nil { + t.Fatal(err) + } + } + + sentinel := sha256.Sum256([]byte("sentinel")) + _, err := db.ExecContext(context.Background(), - "UPDATE files SET head = 'sentinel' WHERE path = ?", []byte(a)) + "UPDATE files SET head = ? WHERE path = ?", sentinel[:], []byte(a)) if err != nil { t.Fatal(err) } st = syncTree(t, db, dir) if st != (scanStats{walked: 2, unchanged: 2}) { - t.Fatalf("rescan stats = %+v, want 2 unchanged", st) + t.Fatalf("rescan stats = %+v, want 2 unchanged and none skipped", + st) } - if r := recordByPath(t, dbRecords(t, db), a); r.head != "sentinel" { - t.Fatalf("head = %q, want sentinel (file must not be re-read)", - r.head) + if r := recordByPath(t, dbRecords(t, db), a); r.head != sentinel { + t.Fatalf("head = %x, want the sentinel %x (file must not be re-read)", + r.head, sentinel) } } @@ -1721,91 +1366,104 @@ func assertWholeFileHashed(t *testing.T, db *sql.DB, path string, t.Helper() r := recordByPath(t, dbRecords(t, db), path) - if want := hexSum(data); r.head != want || r.tail != want || + if want := sha256.Sum256(data); r.head != want || r.tail != want || r.content != want { - t.Fatalf("head, tail, content = %q, %q, %q, want %q for each", + t.Fatalf("head, tail, content = %x, %x, %x, want %x for each", r.head, r.tail, r.content, want) } } +// TestScanStoresRawHashes checks what the database holds for a file +// larger than both end windows, whose three hashes all differ: head, +// tail and content are each a BLOB of the 32 bytes of the SHA-256 the +// scan computed. +func TestScanStoresRawHashes(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + db := openTestDB(t) + data := pattern(3, 3*headTailWindow) + p := writeFile(t, dir, "f.bin", data) + + syncTree(t, db, dir) + + var ( + blobs bool + head, tail, content []byte + ) + + err := db.QueryRowContext(t.Context(), + "SELECT typeof(head) = 'blob' AND typeof(tail) = 'blob' AND "+ + "typeof(content) = 'blob', head, tail, content "+ + "FROM files WHERE path = ?", []byte(p)). + Scan(&blobs, &head, &tail, &content) + if err != nil { + t.Fatal(err) + } + + if !blobs { + t.Error("head, tail and content are not all BLOBs") + } + + wantHead := sha256.Sum256(data[:headTailWindow]) + wantTail := sha256.Sum256(data[len(data)-headTailWindow:]) + wantContent := sha256.Sum256(data) + + if !bytes.Equal(head, wantHead[:]) || !bytes.Equal(tail, wantTail[:]) || + !bytes.Equal(content, wantContent[:]) { + t.Errorf("head, tail, content = %x, %x, %x, want %x, %x, %x", + head, tail, content, wantHead, wantTail, wantContent) + } +} + // TestSyncScanSameSecondRewrite rewrites a file in place at the same // size with an mtime later in the same second as the recorded one: the -// next scan must notice the change and re-hash the file. +// next scan must notice the change and re-hash the file. The operand is +// the file's directory, or the file itself, which scan stats without +// reading its directory. func TestSyncScanSameSecondRewrite(t *testing.T) { t.Parallel() - dir := t.TempDir() - db := openTestDB(t) - a := writeFile(t, dir, "a.bin", pattern(1, 500)) + for _, fileOperand := range []bool{false, true} { + t.Run(fmt.Sprintf("file operand %v", fileOperand), func(t *testing.T) { + t.Parallel() - // b.bin shares the size of a.bin, so a.bin is hashed. - writeFile(t, dir, "b.bin", pattern(2, 500)) + dir := t.TempDir() + db := openTestDB(t) + a := writeFile(t, dir, "a.bin", pattern(1, 500)) - first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC) + operand := dir + if fileOperand { + operand = a + } - err := os.Chtimes(a, first, first) - if err != nil { - t.Fatal(err) + first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC) + + err := os.Chtimes(a, first, first) + if err != nil { + t.Fatal(err) + } + + syncTree(t, db, operand) + + rewritten := pattern(3, 500) + writeFile(t, dir, "a.bin", rewritten) + + later := first.Add(500 * time.Millisecond) + + err = os.Chtimes(a, later, later) + if err != nil { + t.Fatal(err) + } + + st := syncTree(t, db, operand) + if st != (scanStats{walked: 1, updated: 1}) { + t.Fatalf("rescan stats = %+v, want 1 updated", st) + } + + assertWholeFileHashed(t, db, a, rewritten) + }) } - - syncTree(t, db, dir) - - rewritten := pattern(3, 500) - writeFile(t, dir, "a.bin", rewritten) - - later := first.Add(500 * time.Millisecond) - - err = os.Chtimes(a, later, later) - if err != nil { - t.Fatal(err) - } - - st := syncTree(t, db, dir) - if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st) - } - - assertWholeFileHashed(t, db, a, rewritten) -} - -// TestSyncScanOperandSameSecondRewrite is TestSyncScanSameSecondRewrite -// for files given to scan as operands, which scan stats without reading -// their directory. -func TestSyncScanOperandSameSecondRewrite(t *testing.T) { - t.Parallel() - - dir := t.TempDir() - db := openTestDB(t) - a := writeFile(t, dir, "a.bin", pattern(1, 500)) - - // b.bin shares the size of a.bin, so a.bin is hashed. - b := writeFile(t, dir, "b.bin", pattern(2, 500)) - - first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC) - - err := os.Chtimes(a, first, first) - if err != nil { - t.Fatal(err) - } - - syncTree(t, db, a, b) - - rewritten := pattern(3, 500) - writeFile(t, dir, "a.bin", rewritten) - - later := first.Add(500 * time.Millisecond) - - err = os.Chtimes(a, later, later) - if err != nil { - t.Fatal(err) - } - - st := syncTree(t, db, a, b) - if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st) - } - - assertWholeFileHashed(t, db, a, rewritten) } // TestSyncScanRewriteAfter2262 runs assertLateRewriteRehashed with an @@ -1836,9 +1494,6 @@ func assertLateRewriteRehashed(t *testing.T, late time.Time) { db := openTestDB(t) a := writeFile(t, dir, "a.bin", pattern(1, 500)) - // b.bin shares the size of a.bin, so a.bin is hashed. - writeFile(t, dir, "b.bin", pattern(2, 500)) - syncTree(t, db, dir) rewritten := pattern(3, 500) @@ -1867,8 +1522,8 @@ func assertLateRewriteRehashed(t *testing.T, late time.Time) { } st := syncTree(t, db, dir) - if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st) + if st != (scanStats{walked: 1, updated: 1}) { + t.Fatalf("rescan stats = %+v, want 1 updated", st) } assertWholeFileHashed(t, db, a, rewritten) @@ -2017,88 +1672,19 @@ func TestSyncScanOverlappingRoots(t *testing.T) { } } -func TestScanSkipsUniqueSizes(t *testing.T) { +// TestScanHashesUniqueSize checks that a file whose size no other file +// shares is hashed like every other file. +func TestScanHashesUniqueSize(t *testing.T) { t.Parallel() dir := t.TempDir() db := openTestDB(t) - a := writeFile(t, dir, "a.bin", pattern(1, 500)) + data := pattern(1, 500) + a := writeFile(t, dir, "a.bin", data) - writeFile(t, dir, "b.bin", pattern(2, 600)) + syncTree(t, db, dir) - // Neither size is shared, so neither file is read: both records - // are written without hashes and no duplicates are reported. - st := syncTree(t, db, dir) - if st != (scanStats{walked: 2, added: 2}) { - t.Fatalf("stats = %+v, want 2 added", st) - } - - recs := dbRecords(t, db) - for _, r := range recs { - if r.head != "" || r.tail != "" || r.content != "" { - t.Errorf("%s: head = %q tail = %q content = %q, want unhashed", - r.path, r.head, r.tail, r.content) - } - } - - if groups := dupeGroups(t, db); len(groups) != 0 { - t.Fatalf("groups = %+v, want none from unhashed records", groups) - } - - // A new same-size file makes 500 a shared size: the next scan - // hashes both the new file and the previously unhashed unchanged - // one, and they group as duplicates. - c := writeFile(t, dir, "c.bin", pattern(1, 500)) - - st = syncTree(t, db, dir) - if st != (scanStats{walked: 3, added: 1, updated: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 added 1 updated 1 unchanged", - st) - } - - groups := dupeGroups(t, db) - if len(groups) != 1 { - t.Fatalf("groups = %+v, want the a/c pair", groups) - } - - if want := []string{a, c}; !slices.Equal(groups[0].paths, want) { - t.Fatalf("group paths = %q, want %q", groups[0].paths, want) - } -} - -func TestTreesUnhashedNeverEqual(t *testing.T) { - t.Parallel() - - // Two trees identical except for same-name, same-size files without - // a content hash must not compare equal: their content is unknown. - // That holds for unhashed files (possible when the trees were - // scanned separately) and for files of headTailMin or more that - // have only a head and tail. - sum := hexSum(pattern(1, 100)) - shared := []scanRec{ - {path: "/x/t1/f1", size: 100, head: sum, tail: sum, content: sum}, - {path: "/x/t2/f1", size: 100, head: sum, tail: sum, content: sum}, - } - - cases := map[string][]scanRec{ - "unhashed": { - {path: "/x/t1/u", size: 50}, - {path: "/x/t2/u", size: 50}, - }, - "head and tail only": { - {path: "/x/t1/u", size: headTailMin, head: "h", tail: "t"}, - {path: "/x/t2/u", size: headTailMin, head: "h", tail: "t"}, - }, - } - - for name, unknown := range cases { - super, dirs := treeOf(t, append(slices.Clone(shared), unknown...)) - - if tg := collectTreeGroups(dirs, super, nil); len(tg) != 0 { - t.Errorf("%s: tree groups = %d, want 0 (the files may differ)", - name, len(tg)) - } - } + assertWholeFileHashed(t, db, a, data) } func TestScanHardlinksReadOnce(t *testing.T) { @@ -2125,7 +1711,8 @@ func TestScanHardlinksReadOnce(t *testing.T) { ra := recordByPath(t, recs, a) rb := recordByPath(t, recs, b) - if ra.head == "" || ra.head != rb.head || ra.tail != rb.tail { + if ra.head == [sha256.Size]byte{} || ra.head != rb.head || + ra.tail != rb.tail { t.Fatalf("hardlink hashes differ: %+v vs %+v", ra, rb) } @@ -2198,12 +1785,54 @@ func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) { if got := stderr(); strings.Count(got, warning) != 1 { t.Errorf("stderr = %q, want %q once", got, warning) } +} - // The content phase opens files the same way. - _, _, content, err := hashContentOnly(p, headTailMin) - if content != "" || !errors.Is(err, errNotRegular) { - t.Errorf("hashContentOnly on a FIFO = %q, %v; want no hash and %v", - content, err, errNotRegular) +// TestHashPhasePassesOverRemovedFiles removes two walked files before +// the hash phase reads them: one with no other path, and the first of +// two hard links to one file. Neither is an error. The phase prints +// nothing but its progress, counts nothing as skipped, and records only +// the hard link still there. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestHashPhasePassesOverRemovedFiles(t *testing.T) { + stderr := captureStderr(t) + dir := t.TempDir() + alone := writeFile(t, dir, "alone.bin", pattern(1, 300)) + a := writeFile(t, dir, "a.bin", pattern(2, 300)) + b := filepath.Join(dir, "b.bin") + + err := os.Link(a, b) + if err != nil { + t.Fatal(err) + } + + walked, _ := collectWalk(t, []string{dir}, false, 1) + + // a.bin sorts before b.bin, so the run of the two reads it first. + for _, p := range []string{alone, a} { + err = os.Remove(p) + if err != nil { + t.Fatal(err) + } + } + + s := &scanState{db: openTestDB(t), toHash: walked} + + err = s.hashPhase(t.Context(), 1) + if err != nil { + t.Fatalf("hashPhase = %v, want nil", err) + } + + if s.st.skipped != 0 || len(s.batch) != 1 || s.batch[0].path != b { + t.Errorf("skipped = %d, records %+v, want none skipped and a "+ + "record for %s alone", s.st.skipped, s.batch, b) + } + + for line := range strings.Lines(stderr()) { + if !strings.HasPrefix(line, scanPhase("hash")+": ") { + t.Errorf("stderr holds %q, want the hash phase's progress alone", + line) + } } } diff --git a/trees.go b/trees.go index 77275a5..1d2e689 100644 --- a/trees.go +++ b/trees.go @@ -286,18 +286,9 @@ func (b *treeBuilder) finish() (*treeNode, []*treeNode) { // name and its signature (size, head, tail, content); mtime is // excluded. func fileEntry(name string, r scanRec) string { - content := r.content - - // A record without a content hash has unknown content (README - // "Database"): give it a signature no other file can share, so - // trees containing it never compare equal. Real hashes are hex, so - // the NUL-prefixed form cannot collide. - if content == "" { - content = "unhashed\x00" + r.path - } - return "f\x00" + name + "\x00" + strconv.FormatInt(r.size, 10) + - "\x00" + r.head + "\x00" + r.tail + "\x00" + content + "\x00" + string(r.head[:]) + "\x00" + string(r.tail[:]) + + "\x00" + string(r.content[:]) } // collectTreeGroups groups directories by digest, counting each one on @@ -347,7 +338,8 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode, // digest is the SHA-256 of its child entries — files serialized with // name and signature, subdirectories with name and recursive digest — // sorted byte-lexicographically. Filenames cannot contain NUL or "/", -// so NUL delimiters are unambiguous. +// and every hash and digest is 32 bytes, so NUL delimiters are +// unambiguous. func (n *treeNode) computeDigest() { slices.Sort(n.entries) diff --git a/trees_test.go b/trees_test.go index 9249aa0..540c117 100644 --- a/trees_test.go +++ b/trees_test.go @@ -10,24 +10,17 @@ import ( "time" ) -// Signature hashes shared by the smoke-test records. -const ( - f1Head = "f1h" - f1Tail = "f1t" - f1Content = "f1c" - f2Head = "f2h" - f2Tail = "f2t" - f2Content = "f2c" -) - // treesOpenLine is the plain line of trees' open phase when it opens // the database at once; it prints it as the phase starts and as it ends. const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" // smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and // /d/t2 are identical, /d/t3 differs from them only by one filename. func smokeTreeRecs() []scanRec { + f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c") + f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c") + return []scanRec{ {size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"}, {size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"}, @@ -153,13 +146,14 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) { for _, top := range []string{"/a", "/c"} { for _, p := range []string{"/b", "/b-x/f", "/b.txt", "/b/f"} { - content := "c" + content := testHash("c") if p == "/b-x/f" { - content = "other" + content = testHash("other") } recs = append(recs, scanRec{ - size: 1, head: "h", tail: "t", content: content, path: top + p, + size: 1, head: testHash("h"), tail: testHash("t"), + content: content, path: top + p, }) } } @@ -242,11 +236,11 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) { "trees: database " + path + " (from " + databaseEnv + ")", treesOpenLine, treesOpenLine, - "[2/4] read: counting the records, elapsed 0s, eta unknown", + "[2/4] read: counting the records, elapsed 0s, remain unknown", "[2/4] read: [6/6] 100%, ", - "[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown", + "[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown", "[3/4] group: [8/8] 100%, ", - "[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown", + "[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown", "[4/4] write: [1/1] 100%, ", "trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ", } @@ -332,11 +326,10 @@ func TestTreeDigests(t *testing.T) { func TestTreeDigestContentSensitivity(t *testing.T) { t.Parallel() - const sharedTail = "same" - + same, c := testHash("same"), testHash("c") recs := []scanRec{ - {size: 10, head: sharedTail, tail: sharedTail, content: "c", path: "/r/a/f"}, - {size: 10, head: "DIFF", tail: sharedTail, content: "c", path: "/r/b/f"}, + {size: 10, head: same, tail: same, content: c, path: "/r/a/f"}, + {size: 10, head: testHash("DIFF"), tail: same, content: c, path: "/r/b/f"}, } _, dirs := treeOf(t, recs) @@ -397,9 +390,10 @@ func TestCollectTreeGroupsSiblings(t *testing.T) { // Identical sibling dirs share a parent, so their group cannot be // implied by a parent group and must be reported. + h, tl, c := testHash("h"), testHash("t"), testHash("c") recs := []scanRec{ - {size: 10, head: "h", tail: "t", content: "c", path: "/p/x1/f"}, - {size: 10, head: "h", tail: "t", content: "c", path: "/p/x2/f"}, + {size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"}, + {size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"}, } super, dirs := treeOf(t, recs) @@ -418,10 +412,11 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) { // /p/a and /q/b contain identical x subtrees, but /p/a has an // extra file, so the parents' digests differ and the x group must // be reported. + h, tl, c, e := testHash("h"), testHash("t"), testHash("c"), testHash("e") recs := []scanRec{ - {size: 10, head: "h", tail: "t", content: "c", path: "/p/a/x/f"}, - {size: 99, head: "e", tail: "e", content: "e", path: "/p/a/extra"}, - {size: 10, head: "h", tail: "t", content: "c", path: "/q/b/x/f"}, + {size: 10, head: h, tail: tl, content: c, path: "/p/a/x/f"}, + {size: 99, head: e, tail: e, content: e, path: "/p/a/extra"}, + {size: 10, head: h, tail: tl, content: c, path: "/q/b/x/f"}, } super, dirs := treeOf(t, recs)