diff --git a/README.md b/README.md index 4aaca1a..1fdd3e7 100644 --- a/README.md +++ b/README.md @@ -2,25 +2,28 @@ ## Description -`sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that -quickly identifies _candidate_ duplicate files — and, ultimately, entire -duplicate directory trees — across very large filesystems without reading every -byte of every file. Files are considered duplicates when their sizes are equal -and they agree on a short ladder of hashes. A file under 10 MiB is hashed in -full and compared directly. A larger file is gated first on the SHA-256 of its -first 64 KiB and of its last 64 KiB, and only when its size and both of those -match another file's is it read for a content hash to compare — the SHA-256 of -the whole file when it is under 50 MiB, or of gigabyte-spaced 1 MiB samples when -it is 50 MiB or larger. Below 50 MiB the content hash is proof of identical -content; at or above 50 MiB it is a strong candidate signal rather than proof, -because the gaps between samples are never read. The intended use is finding -duplicate downloads and duplicated directory trees on multi-terabyte ZFS servers -where reading every byte of every file is prohibitively expensive. `scan` -maintains a persistent SQLite database of file signatures that survives between -runs, so it can be run from cron and the reports can be generated at any time -from the most recent scan. +`sfdupes` is an MIT-licensed Go command-line tool by +[@sneak](https://sneak.berlin) that finds candidate duplicate files, and whole +duplicate directory trees, on large filesystems. It does this without reading +every byte of every file. -This README is the complete and authoritative specification. +Two files count as duplicates when they have the same size and the same three +SHA-256 hashes. A hash is a short fingerprint of a stretch of bytes. Two +stretches with the same fingerprint are, in practice, identical. Two of the +hashes cover the first and the last 64 KiB of the file. The third covers the +whole file when the file is under 100 MiB. A match then proves identical +content. At 100 MiB and above, the third hash covers only 1 MiB at each +gigabyte. A large file is never read in full. A match between two large files is +a strong candidate, not proof. + +`sfdupes` is meant for multi-terabyte servers, where reading every byte of every +file takes too long. It has five subcommands. `scan` walks the filesystem and +records the files it finds in an SQLite database. SQLite keeps a whole database +in one file. The database is kept between runs. A rescan reads only new and +changed files. That makes `scan` cheap enough to run every day from `cron`, the +system's job scheduler. `report`, `trees` and `info` read the database alone and +can run at any time. `dwim` scans `/` and then prints the `trees` report, for a +first look at a new machine with one command. ## Getting Started @@ -30,60 +33,54 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite" ./sfdupes scan /srv ./sfdupes report > dupes.tsv ./sfdupes trees > dupetrees.tsv +./sfdupes info ``` -`scan` walks one or more filesystem trees and maintains one database record per -regular file (path, size, mtime, head hash, tail hash, content hash). The -database persists between runs; a rescan only hashes files that are new or -changed, or that may have gained a duplicate since the last scan, and removes -records for files that no longer exist. `report` reads the database and prints -the file-level duplicates report. `trees` reads the same database and prints the -duplicate-tree report. `info` prints the database's details: its path, the sizes -of its files and the number of its records. A missing/invalid subcommand — or a -`scan` invocation with no `PATH` operand — prints a usage message and exits 2. +`scan` walks the trees named by its operands, the paths given after `scan` on +the command line. The walk is the listing of every directory under them. `scan` +keeps one database record per regular file. A regular file is an ordinary file, +as opposed to a directory, a symlink or a device node. A symlink is a file that +holds the path of another file and stands in for it. A device node is a file +that stands for a piece of hardware, such as a disk or a terminal. `report` +prints the file-level duplicates. `trees` prints the duplicate directory trees. +`info` prints the database's path, size and record count. -The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere -by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan` -cron job, with the reporting commands run interactively whenever needed; their -results are as fresh as the last completed scan. +The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names +another file. The first `scan` creates it. + +`report`, `trees` and `info` write their data to stdout. Progress, warnings and +errors go to stderr. stdout and stderr are a program's two output streams. The +shell shows both on the terminal, and `>` redirects stdout alone. ### Install -With Go installed, this builds and installs the newest release: +With Go installed, this builds and installs the newest tagged release: ```sh go install sneak.berlin/go/sfdupes@latest ``` -Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time -`main` moves and its tip passes `script/cibuild`, that tip is tagged with the -next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build. -`@main` installs the tip before it is tagged: +`@main` installs the tip of the `main` branch instead: ```sh go install sneak.berlin/go/sfdupes@main ``` -`v0.0.1` is an early build without a database. It does not work as this README -describes, and `go.mod` retracts it. +The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. -The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To -see which version an installed binary is: +`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from +`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is +a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has +one. Otherwise it is a pseudo-version, Go's version string for an untagged +commit. That string ends in the commit's hash. A binary built from a clone +carries the output of `git describe --tags --always --dirty` in that clone, such +as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To +see everything Go recorded in an installed binary: ```sh -sfdupes --version go version -m "$(command -v sfdupes)" ``` -`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`, -`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`, -or for `@main` a pseudo-version ending in the first 12 characters of the -commit's hash. A binary built from a clone or into the Docker image carries the -git tag or commit it was built from instead, and `VERSION` is `dev` only when -the binary carries no version at all. `go version -m` prints what Go recorded in -the binary; its `mod` line names the module's version. Binaries from `v0.0.2` -and earlier print `dev`, so check those with `go version -m`. - From a clone, `make build` writes the binary to `./sfdupes`: ```sh @@ -92,14 +89,16 @@ cd sfdupes make build ``` -Copy the binary to `/usr/local/bin` for the cron job below. +Copy the binary to `/usr/local/bin` for the `cron` job below. -`make docker` builds the Docker image, tagged `sfdupes`, after running the tests -and the linter (see "Build"). The image runs `sfdupes` as root with the database -at its default path, so a bind mount of `/var/lib/sfdupes` keeps the database -between runs. Mount the scanned tree at the same path inside the container as on -the host; read-only is enough. The database records paths as the container sees -them, so the reports then name the host's paths. +`make docker` builds a Docker image tagged `sfdupes`. The build first runs the +tests and the linter, a checker for common mistakes in Go code (see "Build"). +Inside the image `sfdupes` runs as root, with the database at its default path. +Each `-v` flag below makes a bind mount, which shows a host directory at a path +inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between +runs. Mount the tree to scan at the same path inside the container as on the +host. Read-only access is enough. The database then records the host's paths, +and so do the reports. ```sh make docker @@ -108,55 +107,64 @@ docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv ``` -### Daily scan from cron +### Daily scan from `cron` -Run `scan` as root, so that it can read every file: a path it cannot read is -skipped with a warning and loses its database record (see "Rules for the walk"). -As a file `/etc/cron.d/sfdupes`: +Run `scan` as root, so that it can read every file. `scan` warns about a +directory it cannot read, and about a new or changed file it cannot read. It +skips the path and deletes the records at or below it (see "Rules for the +walk"). As the file `/etc/cron.d/sfdupes`: ``` 30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log ``` -- The database is `/var/lib/sfdupes/db.sqlite`, created with its directory by - the first scan. To keep it elsewhere, set - `SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line. +- The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep + the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` before the + command on the same line. - `scan` writes nothing to stdout. Its stderr, appended here to - `/var/log/sfdupes.log`, holds a first line naming the database (see - "Database"), plain progress lines naming each phase and its place among the - five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it - runs, and one as it ends), a warning for each path it skips, a line for each - default exclude its walk does not enter and for each path it leaves out - because it is not on a persistent filesystem (see "Rules for the walk"), and - the summary line (see "Progress" and "`scan` mode"). The log grows with every - scan; rotate it like any other. -- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A - scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the - reason among the last lines of the log; `tail` prints them, and cron mails - them to root if the host can send mail. -- A scan still running when the next one starts carries on. The new one fails at - once, and the lines cron mails include + `/var/log/sfdupes.log`, holds: + - one line naming the database; + - progress lines, as described under "Progress"; + - a warning for each path it skips; + - a line for each system or temporary directory the walk leaves out (see + "Rules for the walk"); + - a line for each path left out because its filesystem is one the kernel, + the core of the operating system, makes up or keeps in memory, such as + `tmpfs`; + - the summary line; + - a warning when a report still has the database open as the scan ends (see + "Concurrent reads"). +- The log grows with every scan. Rotate it like any other log. +- Skipped paths do not fail a scan. The scan exits 0 all the same, and `cron` + sends no mail. +- A scan that fails exits 1, with the reason among the last lines of the log. + `tail` prints those lines, and `cron` mails them to root if the host can send + mail. +- A running scan carries on when the next one starts. The new one fails at once, + and the lines that `cron` mails include `sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`. -- `report`, `trees` and `info` need only read access to the database (see - "Database"). Under the usual umask of `022` the first scan creates it readable - by every user, so an unprivileged user can run them against root's database. +- `report`, `trees` and `info` need only read access to the database. The umask + is the setting that removes permissions from the files a process creates. + Under the usual umask of `022`, the first scan creates the database readable + by every user. An unprivileged user can then run those three commands against + root's database. ### Reading the reports -Each row of `report` names two copies of one file, and each row of `trees` two -copies of one directory tree (see "Report output format" and "Trees output -format"). In a group of copies, the path that sorts first byte by byte is -`first` and every other path is a `dupe` of it. `first` says nothing about which -copy is the original or the oldest; which copy to keep is your choice. +Each row of `report` names two copies of one file. Each row of `trees` names two +copies of one directory tree. Within a group of copies, the path that sorts +first byte by byte is `first`. Every other path in the group is a `dupe` of it. +`first` says nothing about which copy is the original or the oldest. Which copy +to keep is your choice. A row is a candidate, not proof: -- The reports read only the database, so they show the files as of the last - scan; a file may have changed or gone since. -- A file of 50 MiB or more is compared only on samples of its content (see +- The reports show each file as it was when a scan last recorded it. A file may + have changed or gone since. +- A file of 100 MiB or more is compared only on samples of its content (see "Duplicate detection"). -- Paths that are hard links to one file are listed as duplicates, but they share - their data, so removing one frees nothing. +- Two paths that are hard links, two names for one file, are listed as + duplicates. They share their data, so removing one frees nothing. Compare a pair byte for byte before removing either copy. For the row `/srv/a/big.iso`, `/srv/b/big-copy.iso`, `4294967296`: @@ -166,590 +174,516 @@ cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical [ /srv/a/big.iso -ef /srv/b/big-copy.iso ] && echo "hard links" ``` -`cmp` prints nothing and exits 0 only when every byte matches, and otherwise -reports where the files differ. The second line prints `hard links` when the two -paths are the same file, so removing either frees nothing. +`cmp` prints nothing and exits 0 when every byte matches. Otherwise it reports +where the files differ. The second line prints `hard links` when both paths are +the same file. -A path holding a backslash, tab, newline or carriage return is escaped in the -reports (see "Report output format"). Undo the escapes before using it. -`printf '%b'` does exactly that, because every backslash in an escaped path -starts one of the four escapes. Command substitution drops trailing newlines, so -print an `x` after the path and remove it afterwards, or a path that ends in a -newline names a different file: +A path holding a special character, such as a tab, is escaped in the reports. +The character is written as a backslash and one more character, `\t` for a tab +(see "Report output format"). Undo the escapes before using such a path. +`printf '%b'` does that, because every backslash in an escaped path starts an +escape. The shell drops trailing newlines from the output of `$(...)`. To keep a +newline at the end of a path, print an `x` after the path and remove it +afterwards: ```sh p="$(printf '%bx' '/srv/a/tab\tname.txt')"; p="${p%x}" cmp "$p" /srv/b/tab-copy.txt ``` -Check a `trees` row with `diff -r`, which compares the two trees file by file -and also names anything present in only one of them, such as an empty directory -or a symlink, which `trees` does not see. +Check a `trees` row with `diff -r`. It compares the two trees file by file. It +also names anything present in only one of them, such as an empty directory or a +symlink. `trees` does not see those. ## Rationale -Duplicate finders that hash entire files do not scale to the target environment: -~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under -resilver). sfdupes spends disk I/O only on files whose size at least one other -file shares, since a size-unique file cannot be a duplicate. Of those, a file -under 10 MiB is read in full; a larger one has its cheap end windows read first, -and is read for a content hash only when its size and both end windows match -another file's — the whole file below 50 MiB, but only gigabyte-spaced samples -at or above 50 MiB, so the largest files are never read in full. This keeps a -full-filesystem sweep tractable, and the signatures are kept in a persistent -database, so the expensive filesystem pass is incremental: a rescan re-hashes -only files whose recorded mtime or size changed, plus — for its content hash — a -file of 10 MiB or more whose size and end windows have come to match another -file's. All analysis happens offline from the database alone. The end goal is -not individual files but whole duplicated trees — duplicate extractions, -duplicate downloads, copied project trees — which an operator can consider -removing as a unit. +Duplicate finders that hash every file in full do not scale to the target +environment. That is about ten million files and 150 TB. The disks may be slow +or busy. A ZFS storage pool, a set of disks managed as one filesystem, is busy +while it rebuilds a disk. + +`sfdupes` reads a file under 100 MiB in full. It reads a larger file only at its +two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it +makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB +of reads, at the price of a comparison that is not exact. + +The hashes live in a database that is kept between runs. A rescan reads only new +files, and files whose size changed or whose modification time (mtime) is newer +than recorded. All analysis then works from the database alone. + +The goal is whole duplicated trees, not only single files. A download extracted +twice, or a project copied once too often, is such a tree. It can be removed as +a unit. ## Design -Goals, in order: +### Goals -1. **Find whole duplicate trees, not just files.** The end goal is to identify - places where the exact same set of files and directories exists at two or - more paths (duplicate extractions, duplicate downloads, copied project - trees), so the operator can consider removing an entire subtree at once. - File-level duplicate detection is the foundation; tree-level detection is - built on top of it. -2. **Spend I/O in proportion to duplicate likelihood.** Only files whose size - at least one other file shares are read at all — a size-unique file cannot - be a duplicate. Those are compared by the ladder in "Duplicate detection" - below: a file under 10 MiB is hashed in full, while a larger file is gated - on cheap 64 KiB end windows first, and gets a content hash only when its - size and both end windows match another file's. That hash reads the whole - file below 50 MiB but only gigabyte-spaced 1 MiB samples at or above it, so - the very largest files are still never read in full. Scale target: tens of - millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool - under resilver). Holding one small record (path, size, mtime) per file in - memory during a scan is acceptable; holding every file's hashes is not (they - stay in the database). The reporting commands do not hold every file's - hashes either: `report` reads the records in signature order and holds the - duplicate paths of one size at a time, so its memory grows with the - duplicates of the size that has the most of them, and `trees` reads the - records in path order and keeps each directory's path, digest and totals, - plus the hashes of only the files in the directories holding the record - being read, so its memory grows with the number of directories and with the - size of the largest directory. -3. **Scan incrementally, analyze offline.** The expensive filesystem scan - maintains a persistent database; an unchanged file is never read again on a - rescan, except to compute its content hash once a file of 10 MiB or more - comes to match another on size and both end windows. All analysis (`report`, - `trees`) works from the database alone and must never touch the scanned - filesystem again. `scan` is designed to be cronned; the reports run at any - time against the last completed scan. -4. **Clean stream separation.** Everything on stdout is machine-readable data. - All progress, warnings, summaries, and help and usage text go to stderr. - Never mix them. +In order: -### Constraints - -- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`. -- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, **one - progress-bar library** (`github.com/schollz/progressbar/v3`), - `golang.org/x/term` to tell whether stderr is a terminal, **one SQLite - driver** (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled), and - `golang.org/x/sys` for `flock(2)` (the scan lock, see "Database"). - `github.com/spf13/viper` is permitted if configuration-file support is ever - needed, but is not currently used. No other third-party deps. -- Cross-compilation is not a concern. Builds run with cgo disabled (the - `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go. -- Analysis modes (`report`, `trees`) must be deterministic: identical database - contents, identical output, regardless of the order in which records were - inserted. +1. **Find whole duplicate trees, not just files.** The aim is to find places + where the same set of files and directories exists at two or more paths. A + whole subtree can then be removed at once. Tree-level detection is built on + file-level detection. +2. **Bound the reads per file.** A file is read only when it is new or has + changed. A file under 100 MiB is then read in full, and a larger one only at + its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read + in full. The target is a filesystem of about ten million files and 150 TB, on + disks that may be slow or busy. During a scan, one small record per file is + held in memory. The hashes stay in the database. `report` holds the duplicate + paths of one file size at a time. `trees` holds one entry per directory, plus + the files of the directories it is reading. +3. **Rescan cheaply, analyze from the database.** The scan keeps its database + between runs, and an unchanged file is never read again. `report`, `trees` + and `info` work from the database alone and never touch the scanned + filesystem. `scan` is built to run from `cron`. The reports run at any time, + against what the database holds. +4. **Keep the streams apart.** Everything on stdout is machine-readable data. + Everything for a human reader, such as progress and warnings, goes to stderr. ### Subcommands -Four subcommands, all implemented: - -1. `scan` — walk the filesystem and synchronize the database: one signature - record per regular file. -2. `report` — file-level duplicate report from the database. -3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy - from the database records, compute a Merkle-style digest per directory, and - report maximal groups of identical trees. -4. `info` — the database's details: its path, the sizes of its files, its - schema version and the counts of its records. +- `scan` walks the filesystem and updates the database to match it, one record + per regular file. +- `report` prints the file-level duplicate report from the database. +- `trees` prints the duplicate-tree report from the database. +- `info` prints the database's details. +- `dwim` scans `/` with the default excludes, then prints the duplicate-tree + report. ``` sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... sfdupes report > dupes.tsv sfdupes trees > dupetrees.tsv sfdupes info +sfdupes dwim sfdupes --version sfdupes [command] --help ``` -`--workers N` sets the size of each `scan` worker pool (default: the number of -CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that -operand's filesystem, and `--no-default-excludes` lets the walk enter the -default excludes, a list of paths; all three are described under "`scan` mode". -`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see -"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`, -alone or after a subcommand, prints the help text to stderr and exits 0, writing -nothing to stdout. +A missing or unknown subcommand is a usage error, a mistake in the command line. +`sfdupes` then prints the usage text, a summary of the commands and their flags, +on stderr and exits 2. -### Database +The reference below is also the program's specification. -All four subcommands operate on a single SQLite database file: +## Non-goals -- Location: the value of the `SFDUPES_DATABASE` environment variable when set - and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no - command-line flag. The path names the file exactly, whatever characters it - holds (`?`, `#` and `%` included); a relative path is relative to the working - directory. -- Each subcommand first prints one line on stderr naming the database it uses, - before anything else it prints (progress displays included), whether or not - stderr is a terminal. The line gives the path with the working directory put - in front of it when it is relative, and otherwise as set (a `..` in it stays, - so the line names the same file), escaped like the paths in a report (see - "Report output format"), and says where the path came from: +- No byte-for-byte compare. +- No deleting of duplicates. +- No replacing a copy with a hard link to another copy. The reports are + advisory. Acting on them is your job. +- The SQLite database is the only stored data. There are no export or import + formats. +- `sfdupes` does not run in the background or watch the filesystem for changes. + Scheduling rescans is `cron`'s job. - ``` - scan: new database /var/lib/sfdupes/db.sqlite (default) - trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE) - ``` +## Reference - The working directory put in front is the one `pwd -P` prints, with its - symlinks resolved. After a `cd` through a symlink to a directory, a path - starting `../` therefore names a file beside the directory the symlink leads - to, as it does for any other program. `scan` says `new database` when no - file is at that path as it starts. `report`, `trees` and `info` print the - line before they open the database, so the error for a missing database - comes after it. +### Settings -- `scan` creates the database (and its parent directory) on first use. `report`, - `trees` and `info` require an existing database; a missing database file is a - fatal error (exit 1) telling the user to run `scan` first. -- Only one `scan` runs against a database at a time. For its whole run, `scan` - holds an exclusive `flock(2)` lock on a lock file beside the database, named - by appending `.lock` to the database's path with symlinks resolved - (`/var/lib/sfdupes/db.sqlite.lock` by default), taken before it walks the - filesystem or opens the database. Every path to one database, through a - symlink to the file or to a directory above it, names the same lock file. Just - before it takes the lock, `scan` creates the database file, empty, if it is - missing (through a symlink at the database path, it creates the symlink's - target), so the lock file is named the same way before the first scan sets up - the database as after. A second `scan` against the same database does not - wait: it fails at once with a one-line error naming the lock file and exits 1, - without walking anything or opening the database, and the running scan carries - on. The lock file is created on first use, open to its owner only, and left in - place: a leftover file blocks nothing, because the lock ends with the process - holding it however it ends, a fatal error or an interrupt included, and - deleting the file while a scan runs would let a second scan start. `report`, - `trees` and `info` never take the lock, so they run during a scan. -- While `scan` runs, the database is in WAL journal mode with a busy timeout, so - running a report while a cron `scan` is in progress is safe. The filesystem is - authoritative; the database is an eventually-consistent reflection of it. - Hashed records are committed in batched transactions while the scan is still - running (keeping the WAL small and letting concurrent reports observe - progress), so a report may see a scan's changes partially applied, and a scan - that dies partway leaves a valid database holding every batch committed so far - (an interrupted scan also commits the batch in progress, see "Error handling - and exit codes"); the next scan skips those records and converges toward the - filesystem. -- `scan` switches the database back to rollback-journal mode when it closes it, - so between scans the database file alone holds the whole database. Each switch - needs the database to itself: a `scan` that starts while a report is still - reading waits for it up to the 10-second busy timeout, then fails; a `scan` - that ends while a report has the database open warns and leaves the database - in WAL mode until the next scan. `report` writes its rows while it reads the - records, so it is still reading while its output is paused (a pager, a stalled - pipe), and a `scan` started then fails after the busy timeout. -- A `scan` writes to the database and the files beside it at these points (its - phases are described under "`scan` mode"): - - As it starts, before the walk: the directory, an empty database file and - the `.lock` file, each only if missing. - - Then, still before the walk: the schema, in a new database, and the `-wal` - and `-shm` files, which stay beside the database while the scan runs. - - While it loads the existing records (`load`) and walks: no records. A - first scan's database holds none, however long its walk has run. - - During the hash phase: the first records, committed 10,000 at a time. - - In the update phase: the rest of the hashed records, the records for - size-unique files, and the deletions. - - In the content phase: content hashes, again committed 10,000 at a time. - - At the end, as it closes the database: the `-wal` and `-shm` files go - away, unless a report still has the database open (see above). -- `report`, `trees` and `info` open the database read-only and need only read - access to the database file, and no write access to its directory. While the - database is in WAL mode they also read the `-wal` and `-shm` files beside it, - which SQLite creates with the database file's permissions. -- Schema (`PRAGMA user_version` is the schema version, currently 1; a database - with any other version is a fatal error. `scan` creates the schema and sets - the version in one transaction, so a first scan stopped while doing so leaves - an empty database the next scan sets up. A database at version 0 that already - has a `files` table was therefore not made by sfdupes; every subcommand - refuses it with an error telling the user to remove the file and rescan): - - ```sql - CREATE TABLE files ( - path BLOB PRIMARY KEY, -- absolute path, raw bytes - size INTEGER NOT NULL, -- bytes, from lstat - mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat - mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999 - head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB - tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB - content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples - ) WITHOUT ROWID; - CREATE INDEX files_signature ON files (size, head, tail, content); - ``` - - Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed - UTF-8. `mtime` and `mtime_nsec` hold the file's mtime to the nanosecond: - `mtime` the whole Unix seconds, rounded down, and `mtime_nsec` the - nanoseconds past that second. Split this way they hold any mtime a - filesystem can record, one before 1678 or after 2262 included, which a - single 64-bit count of nanoseconds cannot. They are used only for change - detection and are not part of the duplicate key. For a file under 10 MiB - `head`, `tail`, and `content` all hold the whole-file hash (that range is - hashed in full, with no end windows); for a larger file `head` and `tail` - hold the first- and last-64 KiB hashes and `content` the whole-file or - sampled hash. All three are empty strings when the file has never been - hashed because its size was unique as of the last scan that covered it. For - a file of 10 MiB or more, `content` stays empty until the content phase of a - scan (see "`scan` mode" below) has read the file. A record with an empty - `content` is never part of a duplicate group, though it still defines the - file for tree reconstruction. The `files_signature` index lets `report` read - the records in signature order without sorting the table. +- `SFDUPES_DATABASE` names the database file. When it is unset or empty, the + database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for + it. The path names the file exactly, whatever characters it holds. A relative + path is relative to the working directory. +- `--workers N` (`scan`) sets how many directories `scan` reads at once while it + lists the trees. It also sets how many files `scan` reads at once while it + hashes them. The default is the number of CPUs. `N` must be at least 1. A + smaller value is a usage error, reported in one line on stderr with exit 2 + before anything is scanned. +- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of + directories on other filesystems, as the same flag does for `du` and `rsync`. + By default the walk also enters directories on other filesystems. +- `--no-default-excludes` (`scan`) lets the walk enter the default excludes. + Those are the system and temporary directories the walk otherwise leaves out, + such as `/var/tmp` on Linux (see "Rules for the walk"). +- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and + exits 0. It writes nothing to stderr. +- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr + and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the + default excludes of the system it runs on, and the filesystem types the walk + never records anything on. ### Duplicate detection -Two files are duplicates only when they agree on every rung of this ladder; a -mismatch at any rung means they are not duplicates. `scan` stores each file's -hashes, and `report` and `trees` group files by the whole signature — size, -`head`, `tail`, and `content` — so the grouping is exactly this ladder applied -across everything scanned into the database, even across separate scans. +Two files are duplicates when they have the same size and the same three hashes. +A file's size and its three hashes together are its signature. `scan` computes +the hashes and stores them in the database. `report` and `trees` group the files +with the same signature, whichever scan recorded them. -1. **Size.** Files of different sizes are never compared. Only files whose size - at least one other file shares are hashed at all. -2. **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed in full - and compared directly, with no separate end-window step — small files are - cheap to read to the last byte, and doing so makes the comparison exact. - `head`, `tail`, and `content` all hold this whole-file SHA-256, so such a - file's signature is decided entirely by its size and its content. -3. **10 MiB and above: head and tail.** For a larger file, the SHA-256 of the - first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a cheap gate that - eliminates most same-size pairs before any bulk reading: the content hash of - the next two rungs is computed only for a file whose size, `head`, and - `tail` match another file's, whether that file is scanned in the same run or - stored by an earlier scan. A stored file that first gains such a match in a - later scan gets its content hash then; until it has one, its `content` is - empty and it is not a duplicate. At 10 MiB and above the two windows never - overlap. -4. **10 MiB and above, content below 50 MiB.** The SHA-256 of the entire file. - Agreement here is proof of identical content (barring a SHA-256 collision). -5. **10 MiB and above, content 50 MiB and above.** A sampled SHA-256: the 1 MiB - window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … while inside the - file, the final window truncated at end of file) is fed, in order, into one - hash. This is **deliberately probabilistic** — the gaps between samples are - never read, so two large files that agree on every sample are reported as - duplicates without being read in full. It is the price of never reading a - 150 GB file end to end. Because size is already part of the signature, only - equal-size files reach this rung, so their sample boundaries always align. +Every record holds three SHA-256 hashes: -`head`, `tail`, and `content` are one column each. A file below 10 MiB and one -at or above it never share a size, and neither do a file below 50 MiB and one at -or above it, so a stored value is never ambiguous between the whole-file, -end-window, and sampled forms. +- `head`, the hash of the file's first 64 KiB; +- `tail`, the hash of the file's last 64 KiB; +- `content`, the content hash described below. + +The content hash depends on the file's size: + +- Under 100 MiB, it is the SHA-256 of the whole file. A match proves identical + content. Two different files with the same SHA-256 are not a practical + concern. +- At 100 MiB and above, it is the SHA-256 of samples. A sample is the 1 MiB that + starts at each gigabyte boundary, the first at offset 0 and the next at 1 GiB. + The samples go into one hash in order. A sample that would run past the end of + the file stops there. + +In a file of 64 KiB or less, each 64 KiB stretch is the whole file. All three +hashes are then the hash of the whole file. An empty file has fixed hashes and +is never opened. + +The sampled hash is not exact. Bytes outside the samples and outside the first +and last 64 KiB are never read. Two large files that differ only in such bytes +are reported as duplicates. That is the price of never reading a 150 GiB file +end to end. + +Only files of the same size are compared, so their samples lie at the same +offsets. A file under 100 MiB and one of 100 MiB or more never share a size. So +a whole-file hash is never compared with a sampled one. + +`scan` reads a file only to hash it, and only when the file is new or has +changed since its record was written. ### `scan` mode -`scan` requires one or more `PATH` operands naming the trees to scan. There is -no default path; invoking `scan` with no operand is a usage error (usage message -on stderr, exit 2). An operand may be a directory or a regular file; an operand -that does not exist is a fatal error (exit 1). Because database records persist -between runs and are keyed by absolute path, each operand is resolved to an -absolute, lexically cleaned path (symlinks are not resolved) before walking, so -results do not depend on the working directory. All operands belong to a single -scan and are enumerated concurrently: every operand seeds the shared walk worker -pool. Overlapping operands are harmless — an operand that duplicates another or -lies under another is dropped before walking, so every file is reached exactly -once and produces one database record. An operand under another one is kept, -though, and walked on its own when any directory from it (from its directory, -for a regular file) up to but not including the operand covering it is a default -exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with -`-x`, is on a different filesystem than that operand, because that operand's -walk does not enter such a directory and so does not reach it. +`scan` takes one or more `PATH` operands naming the trees to scan. There is no +default. An operand may be a directory or a regular file. -An operand that is a symlink (never followed, not even as an operand), socket, -FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints -a one-line warning naming the path and what it is, counts it as skipped, and -drops it from the scanned operands before reading the database. Another operand -beneath it is still scanned. The records stored beneath it are not deleted: they -are treated like any other record outside the scanned operands, including the -content-phase exception below. If it lies under another operand, they are under -that operand instead, and are treated like any other record there. This is not -an error: a scan whose every operand is dropped walks nothing and exits 0. An -operand that is not on a persistent filesystem (see "Rules for the walk") is -dropped the same way, except that the line `scan` prints for it is not a -warning, and it is not counted as skipped. +#### Operands -`scan` synchronizes the database with the filesystem state under the scanned +- `scan` with no operand is a usage error. The usage text goes to stderr, and + the exit code is 2. +- An operand that does not exist is a fatal error, and so is a symlink whose + target does not exist. `scan` then stops before it walks anything, and exits + with code 1. +- Each operand is resolved to its real path before the walk, as `realpath` + prints it. A relative operand is joined to the working directory as `pwd -P` + prints it. Every symlink in the path is then followed. A `..` after a symlink + counts from the directory the symlink leads to. From then on the operand is + that real path. Its files are recorded under it, and warnings name it. +- An operand that is a symlink is scanned at its target's real path. After a + `cd` through a symlink, `scan .` records its files under the real path of the + working directory. +- All operands belong to one scan, and one walk covers them all at once (see + "Concurrency"). +- An operand that repeats another is dropped before the walk. Two operands + repeat each other when their real paths are the same. +- An operand under another operand is dropped the same way, unless one of the + cases below applies. No path is then walked twice. Whether one operand is + under another is judged on their real paths. With `/srv/link` a symlink to a + directory outside `/srv`, `scan /srv /srv/link/sub` walks `sub` on its own, + because the real path of `sub` is not under `/srv`. +- An operand under another is kept, and walked on its own, when the inner + operand or a directory between the two is one of these: + - a default exclude (see "Rules for the walk"); + - with `-x`, a directory on another filesystem than the outer operand. + + It is also kept when a directory between the two is on a filesystem that is + not persistent. Such a filesystem is one the kernel makes up or keeps in + memory (see "Rules for the walk"). The outer operand's walk does not enter + such a directory, so the inner operand is walked on its own. + +- No other case keeps an operand under another. +- An operand that is neither a regular file nor a directory is not scanned. Such + an operand is, for example, a symlink to a device node. `scan` prints a + warning naming the real path and what it is. It counts the operand as skipped + and drops it. The scan keeps the records beneath a dropped operand, unless + another operand covers them. +- An operand at or inside a directory named `.zfs` is never scanned, whether + given alone or under another operand. The check looks at the real path, so a + symlink into a `.zfs` directory is not scanned either. `scan` prints a warning + naming the real path, counts the operand as skipped and drops it. The records + beneath it are kept, unless it lies under another operand. Then they are + deleted, because that operand's walk never enters `.zfs` (see "Rules for the + walk"). +- An operand on a filesystem that is not persistent is dropped too. `scan` + prints a line naming it and its filesystem type, and does not count it as + skipped. +- `scan` prints these warnings, and the line for an operand that is not on a + persistent filesystem, even when the operand repeats another or lies under + another. +- A scan whose every operand is dropped walks nothing and exits 0. + +#### What a scan does to the database + +`scan` brings the database in line with the filesystem under the scanned operands: -- Only a file whose size at least one other file shares is ever read: a - size-unique file cannot be a duplicate, so it is recorded without hashes - (`head`, `tail`, and `content` empty). The size census covers every file - walked this scan plus every database record outside the scanned operands, so a - possible duplicate of a separately scanned tree is still recognized. -- A file not yet in the database is inserted: hashed when its size is shared, - without hashes otherwise. -- A file already in the database is **skipped without reading its contents** - when its lstat size equals the recorded size and its lstat mtime is not newer - than the recorded mtime. This is what makes a daily rescan cheap. Exception: - an unchanged file whose record lacks hashes is hashed — and its record updated - — once its size becomes shared, so hashing deferred by size-uniqueness happens - as soon as it could matter. Likewise, an unchanged file of 10 MiB or more - whose record has no `content` hash is read for one by the content phase below - once its size, `head`, and `tail` match another record's. -- A file whose mtime is newer than recorded, or whose size differs, is processed - as if new: re-hashed, or recorded without hashes, per the shared-size rule. - Change detection compares the mtime to the nanosecond, as finely as the - filesystem records it, so a same-size rewrite counts as a change whenever the - filesystem gives it a later mtime than recorded, even within the same second. -- A database record whose path lies under one of the scanned operands but was - not successfully processed this run is deleted. This removes records for - deleted files. It also removes records for paths that failed to stat or hash - this run: the database only ever contains signatures verified by the most - recent scan that covered them (a subsequent successful scan re-adds such - files). A failure in the content phase below removes nothing: the record is - left as it is. -- Database records outside the scanned operands are untouched, so disjoint trees - can be scanned on different schedules into the same database. The one - exception is the content phase below: a stored file of 10 MiB or more without - a `content` hash is read for one, wherever it lies, once its size, `head`, and - `tail` match another record's. If that file is gone, has changed since its - record was written, or is not on a persistent filesystem, it is not read and - the record is left as it is. -- A record at or below a path the walk left out, because that path is a default - exclude, is not on a persistent filesystem, or, with `-x`, is a directory on - another filesystem than its operand, counts as outside the scanned operands: - it is not deleted, and its size joins the size census. A filesystem mounted - under an operand can therefore be scanned on its own schedule into the same - database. Records below an operand at or inside such a path are not kept this - way, since that operand's own walk verifies them. +- A file not yet in the database is hashed and inserted. +- A file already in the database is not read again when its size and mtime match + its record. The mtime matches when it is not newer than the recorded one. This + is what makes a daily rescan cheap. +- When such a file's size on disk differs from its recorded `disk_size`, `scan` + updates `disk_size` alone. It still does not read the file, and counts it as + unchanged. ZFS, for example, counts a new file's blocks up to seconds after + the write. A block is the unit in which a filesystem allocates disk space. A + file scanned within those seconds gets its real `disk_size` at the next scan. +- `scan` hashes a file again when its size differs or its mtime is newer. The + mtime is compared to the nanosecond, as finely as the filesystem records it. A + same-size rewrite counts as a change whenever the filesystem gives it a later + mtime, even within the same second. +- The scan deletes each record under a scanned operand whose file it could not + verify. That covers deleted files, and paths the scan could not stat or read + this run. To stat a file is to read its details, such as its size, without + opening it. So after a complete scan, the database holds only what that scan + could verify under its operands. A later successful scan adds such files + again. +- Records outside the scanned operands are left alone, so separate trees can be + scanned on different schedules into one database. +- The scan keeps every record at or below a path that the walk leaves out for + one of these reasons: + - it is a default exclude; + - it is not on a persistent filesystem; + - with `-x`, it is a directory on another filesystem than its operand. -`scan` runs **five sequential phases over the whole scan**. Parallelism lives -inside the walk, hash and content phases; batched database writes begin during -the hash phase: + Such records count as outside the scanned operands. So with `-x`, a + filesystem mounted under an operand can be scanned on its own schedule into + the same database. An operand may itself lie at or inside such a path, as + "Operands" describes. Its walk verifies the records below it, so they are + deleted when their files are gone. -1. **load** — read the existing records. Each record under the operands is kept - in memory, by path, with its size, its mtime and whether it has hashes, for - the walk to compare each file with; the size of every other record joins the - size census. -2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently - with the walk worker pool: every operand seeds the shared queue, and each - worker reads one directory at a time, handing discovered subdirectories back - to the queue and running `lstat` on each regular file as it is discovered - (while the directory's metadata is still hot). Sequential directory - enumeration is metadata-latency-bound and takes hours at tens of millions of - files; per-directory parallelism is what makes the walk tractable on large - or busy pools. The walk builds the size census and resolves unchanged - already-hashed files on the fly; every other file is carried to the hash - phase as a (path, size, mtime) record. -3. **hash** — with the census complete, each carried file's size decides its - fate. Size-unique files are never read: new or changed ones are recorded - without hashes in the update phase, unchanged unhashed ones simply keep - their records. Every file with a shared size is hashed by the worker pool as - described in "Duplicate detection" above: a file under 10 MiB in full, which - gives its `head`, `tail`, and `content` alike, and a larger file only in its - end windows, which give its `head` and `tail`; its content hash is left to - the content phase. Zero-length files have constant hashes and are never - opened. Files are hashed in **inode order** (minimizing seeks on spinning - disks), and paths that are hard links to the same inode are **read once**, - all sharing the one result — a hard-link backup farm costs one read per - inode, not per path. The phase total counts actual reads, so progress and - ETA are meaningful. Completed records are committed in batched transactions - **while hashing runs**, so a scan interrupted after hours keeps everything - hashed so far and the next scan resumes cheaply, skipping records already - written. -4. **update** — commit the final partial batch, the hash-less records for - size-unique new and changed files, and the deletions for records the scan - did not verify (vanished files, plus paths that failed to stat or hash). -5. **content** — find every record of 10 MiB or more without a `content` hash - whose size, `head`, and `tail` equal another record's, anywhere in the - database: records from this scan and records stored by earlier scans, inside - or outside the scanned operands. SQLite finds them, so only the records to - be read are kept in memory, never every file's hashes. Every record sharing - their size, `head`, and `tail`, including one that already has a `content` - hash, has its file checked with `lstat` first. A file that is gone, is no - longer a regular file, has changed (a different size, or an mtime newer than - recorded), or is not on a persistent filesystem (see "Rules for the walk") - keeps its record as it is and does not count as a match for the others. Any - other `lstat` error is warned about and counted as skipped, with the same - result. If such a record has no `content` hash, it stays out of duplicate - groups; if it has one, it is still reported until a scan covering its own - tree updates or removes it. The files that pass and have no `content` hash - are read only if at least two of those records pass, so a file whose only - matches are stale costs no read; a file that already has a `content` hash is - never read again. They are read by a worker pool as in the hash phase, in - inode order and once per inode, and their content hashes are committed in - batches. A failed read is warned about and counted as skipped; its record - keeps an empty `content`, so it is not a duplicate, and a later scan tries - again. +#### Phases -Rules for the walk: +`scan` runs four phases in order. Parallel work happens inside the walk and hash +phases. -- Only regular files. Skip directories, symlinks (do not follow, including - symlink operands), sockets, FIFOs, and device nodes. An operand that is a - symlink, socket, FIFO, or device node is dropped as described in "`scan` mode" - above. -- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking - them would list every file once per snapshot), not even when it is an operand; - such an operand is dropped the same way. -- Never record the scan's own database. Once the database and its `.lock` file - are open, `scan` notes the device and inode of the database file, its `-wal` - and `-shm` files and its `.lock` file, and the walk passes over a file with - one of them, however it reaches it: as an operand, below one, or through a - symlinked directory. It prints no line for such a file and does not count it - as skipped. A record that an earlier scan left for one of them is deleted, - like the record of any file the scan no longer verifies. There is no - `-journal` file during the walk, because the database is in WAL mode then (see - "Database"). -- Filesystem boundaries are crossed by default. With `-x` (long form - `--one-file-system`, following the GNU `du`/`rsync` convention), never descend - into a directory on a different filesystem than its `PATH` operand; each - operand is bounded by its own filesystem. The records below such a directory, - unless they lie below an operand at or inside that directory, count as outside - the scanned operands, as described in "`scan` mode" above: they are not - deleted, and their sizes join the size census. -- Never record anything that is not on a persistent filesystem. A filesystem is - not persistent when the kernel makes it up or keeps it in memory: one of type - `proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`, - `debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`, - `mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`, - `rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After - it names the database and before it takes the lock, `scan` lists the mounted - filesystems, from `/proc/self/mountinfo` on Linux and from the system's list - of mounted filesystems on macOS (on any other system it lists none), and notes - the device ID of each one of those types. On Linux that is the device ID the - list gives, so a filesystem that a later mount hides still gives its own. On - macOS it is read with `lstat` on the mount point, and where one filesystem is - mounted over another on the same mount point, the one mounted last counts. If - `scan` cannot list them, it fails (exit 1). The walk then never enters a - directory, and never records a file, on one of those devices, however it - reaches it: as an operand, below one, or through a bind mount, which keeps the - device ID of what it binds. The content phase never reads a file on one of - them either. No flag turns this off. For each operand, directory or file it - leaves out this way, `scan` prints a line on stderr naming it and the type of - its filesystem, as it prints a warning, so it never lands inside a progress - display: +1. `load` reads the existing records. Each record under the operands is kept in + memory by path, with its size, its size on disk and its mtime, for the walk + to compare against. +2. `walk` lists the trees under all operands. Several workers share one queue of + directories to read. A worker is one thread of the program. A thread is one + of several parts of a program that run at the same time and share its memory. + "Concurrency" says how many workers there are. Each worker reads one + directory at a time. It hands the subdirectories it finds back to the queue, + and it stats each regular file as it finds it. The walk compares each file + with its record. An unchanged file is not read again. When its size on disk + has changed, the update phase writes its new `disk_size`. Every new or + changed file is passed to the hash phase. +3. `hash` reads every new or changed file and computes its three hashes. Files + are read in inode order. An inode is the filesystem's own record of a file. + Files with nearby inode numbers usually lie near each other on disk. Reading + them in that order spares a spinning disk from seeking back and forth. Paths + that are hard links to one file are read once and share the result. A backup + tree made of hard links therefore costs one read per file. Completed records + are written to the database 10,000 at a time while hashing runs. So a scan + interrupted after hours keeps everything hashed so far. +4. `update` writes the hashed records not yet written, and the new `disk_size` + of each unchanged file whose size on disk changed. Then it deletes the + records the scan did not verify. + +Reading a tree one directory at a time waits on the disk at every step and takes +hours at millions of files. Reading many directories at once is what makes the +walk practical on large or busy storage pools. + +#### Rules for the walk + +- Only regular files are recorded. Every other kind of file, such as a symlink + or a device node, is passed over. The walk follows no symlink it finds. +- The walk never enters a directory named `.zfs`. Such a directory holds the + filesystem's ZFS snapshots, which are its saved earlier states. Walking it + would list every file once per snapshot. `scan` never starts a walk at or + inside one either (see "Operands"). +- The walk never records the scan's own database files: + - the database file; + - its `.lock` file (see "The scan lock"); + - the `-wal` and `-shm` files SQLite keeps beside it while the scan runs + (see "Concurrent reads"). + + The walk knows them by device ID and inode number. The device ID is the + number by which the kernel tells filesystems apart. The inode number tells + the files of one filesystem apart. So the walk passes over them whatever + path leads to them. The walk follows no symlink it finds. It opens each + operand at its real path, not the path given. An operand given through a + symlink can lead to the database files, and the walk passes over them there + as well. The walk prints no line for the database files and does not count + them as skipped. A record that an earlier scan left for one of them is + deleted. + +- By default the walk enters directories on other filesystems than its + operand's. With `-x`, it never enters a directory on a different filesystem + than its operand. `scan` prints no line for a directory that only `-x` leaves + out, and does not count it as skipped. The records below it are kept, as + described under "What a scan does to the database". +- The walk never records anything on a filesystem that is not persistent. A + filesystem is not persistent when the kernel makes it up or keeps it in + memory. These are the types, as `sfdupes scan --help` lists them on Linux and + macOS: + + ``` + proc, sysfs, devtmpfs, devpts, tmpfs, ramfs + cgroup, cgroup2, debugfs, tracefs, securityfs, pstore + bpf, configfs, fusectl, mqueue, hugetlbfs, efivarfs + binfmt_misc, autofs, nsfs, rpc_pipefs, selinuxfs, devfs + ``` + + `devfs` is the macOS filesystem of `/dev`. The others are Linux names. As it + starts, `scan` lists the mounted filesystems and notes the device ID of each + one of those types. On Linux the list comes from `/proc/self/mountinfo`, + which gives each filesystem's own device ID. On macOS it comes from the + system's mount list. There the device ID is read from the mount point, the + directory the filesystem is mounted on. Where two filesystems share a mount + point on macOS, the one mounted last counts. On any other system the list is + empty. A list that `scan` cannot read is a fatal error, and nothing is + scanned. No flag turns this rule off. + + The walk never enters a directory on one of those devices, and never records + a file on one. That holds however the walk reaches the path, through a bind + mount included. A bind mount shows a directory at a second path, and the + files under it keep their device ID. For each path it leaves out this way, + `scan` prints a line on stderr naming the path and its filesystem type: ``` scan: skipped /dev/shm (tmpfs, not a persistent filesystem) ``` - That line is not a warning, and the path is not counted as skipped. Records - that an earlier scan left at or below it, unless they lie below an operand - at or inside it, count as outside the scanned operands, as described in - "`scan` mode" above: they are not deleted, and their sizes join the size - census. + The path is not counted as skipped. The records below it are kept, as + described under "What a scan does to the database". -- Below an operand, never descend into a default exclude: a path that holds the - kernel's view of the running system, or temporary files, so that - `sfdupes scan /` walks only the files worth comparing. The default excludes - depend on the system, and `sfdupes scan --help` lists those of the system it - runs on: - - On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`. - - On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`, - `/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches - everything in `/System/Volumes/Data` through the firmlinks, so walking it - as well would record each of those files twice. - - On Windows, relative to the root of the drive an operand lies on: the - page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and - `hiberfil.sys`, and the directories `System Volume Information`, - `$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*` - matches one path component, as it does in any default exclude. sfdupes - does not build for Windows yet, so this list is not used. - - On any other system there are none. +- Below an operand, the walk does not enter a default exclude. On Linux, the + default excludes hold temporary files and the kernel's view of the running + system. They depend on the system, and `sfdupes scan --help` lists those of + the system it runs on. On Linux they are: - `--no-default-excludes` turns them all off; what is not on a persistent - filesystem stays left out. An operand at or inside a default exclude is - still walked, so `sfdupes scan /tmp/x` works, unless it is not on a - persistent filesystem. For each default exclude its walk does not enter, - `scan` prints a line on stderr naming it, as it prints a warning: + ``` + /proc, /sys, /dev, /run, /tmp, /var/tmp + ``` + + On macOS they are: + + ``` + /dev, /private/tmp, /private/var/tmp, /private/var/vm + /System/Volumes/VM, /System/Volumes/Data + ``` + + A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already, + through firmlinks. Firmlinks are the links macOS uses to show the data + volume inside the system volume. Walking `/System/Volumes/Data` as well + would record each of those files twice. Any other system has no default + excludes. + + An operand at or inside a default exclude is walked, so + `sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem. + `--no-default-excludes` turns the default excludes off. What is not on a + persistent filesystem stays left out. For each default exclude the walk does + not enter, `scan` prints a line on stderr naming it: ``` scan: skipped /var/tmp (default exclude) ``` - That line is not a warning, and the directory is not counted as skipped. A - default exclude that is not on a persistent filesystem, as `/proc` is, gets - the line for its filesystem instead, and one that is itself an operand gets - no line, since it is walked as that operand. The records below a default - exclude the walk does not enter, unless they lie below an operand at or - inside it, count as outside the scanned operands, as described in "`scan` - mode" above: they are not deleted, and their sizes join the size census. + The directory is not counted as skipped. `scan` prints the filesystem line + instead for a default exclude that is not on a persistent filesystem, such + as `/proc`. One that is itself an operand gets no line, because it is walked + as that operand. The scan keeps the records below a default exclude that the + walk did not enter, as described under "What a scan does to the database". -- On any per-path error (permission denied, file vanished between passes, - unreadable): print a one-line warning to stderr, skip the path, and continue. - Per-file errors never abort the run; the final summary reports how many were - skipped. As specified above, a skipped path that has a database record from an - earlier scan loses that record, unless it failed only in the content phase, or - is an operand dropped before the database was read whose record counts as - outside the scanned operands; an unreadable directory subtree likewise loses - its records (accepted: the database mirrors what the latest scan could - actually verify). +- `scan` skips each of these, with a one-line warning on stderr: + - a directory it cannot read; + - a path it cannot stat; + - a new or changed file it cannot read. -Concurrency: the walk phase (which also stats files), the hash phase, and the -content phase each use a worker pool of `--workers` workers (default -`runtime.NumCPU()`); the walk parallelizes across directories, hashing across -files. `--workers` must be at least 1: a smaller value is a usage error, -reported in one line on stderr with exit 2 before anything is scanned. All three -phases are seek-bound on spinning disks, so raising `--workers` well past the -core count can help on pools with many spindles. The main goroutine owns -partitioning, database writes, and the progress counts, which each progress -display draws from a goroutine of its own; progress display must never block the -workers. + An unchanged file is not opened, so it keeps its record even when it has + become unreadable. The scan continues, and the summary line says how many + paths were skipped. A skipped path that has a record from an earlier scan + loses that record, and so does every path under an unreadable directory. -`scan` writes nothing to stdout. The summary line on stderr reports the files -seen this run broken down by disposition, plus skips: +- A file or directory that disappears while the scan runs is not an error. The + scan prints no line for it and does not count it as skipped. A path gone when + the walk reaches it gets no record, and so does a file gone when the hash + phase opens it. The records that an earlier scan left at or below it are + deleted. + +#### Concurrency + +The walk phase and the hash phase each have `--workers` workers of their own. +The walk works on several directories at once, and the hash phase on several +files at once. On spinning disks both phases wait mostly on seeks, the time the +disk head takes to move. A `--workers` value well above the CPU count can +therefore help on storage pools with many disks. The main thread of the program +owns the database writes and the progress counts. Each progress display draws +from a thread of its own and never blocks the workers. + +#### What `scan` prints + +`scan` writes nothing to stdout. On stderr it prints, in this order: + +- one line naming the database (see "Database"); +- for each dropped operand, in operand order, the warning or the line described + under "Operands". `scan` prints neither for an operand it drops only because + it repeats another or lies under another; +- the progress display of each phase (see "Progress"). `scan` prints each + warning for a skipped path as it happens. It prints a line the same way for + each path left out as a default exclude or as not on a persistent filesystem; +- the summary line; +- a warning when a report still has the database open as the scan ends (see + "Concurrent reads"). + +The summary line reports the files seen this run, by what happened to each, plus +the skips: ``` scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped ``` -(`removed` counts deleted database records, which are not part of the files-seen -total.) +`removed` counts deleted records, which are not part of the files seen. + +#### Stopping a scan + +`scan` stops cleanly on `SIGINT`, the signal Ctrl-C sends, or `SIGTERM`, the +signal `kill` sends by default: + +- The workers stop taking work. Each finishes no more than the directory listing + or the file it is reading. The progress display is finished. +- The records hashed but not yet written are written, so the next scan does not + hash them again. +- Nothing else is written or deleted. Records are deleted only after a complete + walk. So the records under paths that an interrupted walk never reached are + kept. +- The database is closed and the lock released. +- The line `scan: interrupted after N files` goes to stderr. `N` is the number + of files the walk reached. The exit code is 1. + +The next scan reads none of the unchanged files already recorded, and brings the +database up to date as usual. + +After the first signal `scan` stops catching signals. A second one ends it at +once, as any uncaught signal does. The records not yet written are lost, and the +database stays valid. `scan` catches `SIGTERM` even when it inherits it as +ignored. A background job started from a script inherits `SIGINT` as ignored, +and `scan` leaves it ignored. ### `report` mode -`report` reads every record from the database and takes no positional arguments. - -**`report` must never touch the filesystem being analyzed.** It does not stat, -open, or otherwise access any path that appears in the records; its only I/O is -reading the database and writing stdout/stderr. `report` must produce identical -output whether or not the scanned filesystem is still mounted. +`report` reads every record from the database. It takes no positional arguments. +A positional argument is anything on the command line that is not a flag, such +as the paths `scan` takes. `report` never touches the scanned filesystem. It +neither stats nor opens any path in the records. It reads the database and +writes to stdout and stderr, and nothing else. Its output is the same whether or +not the scanned filesystem is still mounted. Processing: -- Records without a `content` hash (see "Database" above) are excluded: their - content is unknown, so they are never reported as duplicates. -- Group the remaining records by the key `(size, head, tail, content)`. +- Records are grouped by signature, the four values + `(size, head, tail, content)`. - Every group with two or more paths is a duplicate group. -- Within each group, sort paths lexicographically (byte order). The first path - is the group's `first`; every other path is a `dupe`. -- Order groups by size descending (biggest reclaimable space first), tie-broken - by `first` path ascending. Output must be fully deterministic for a given - database state. +- Within a group, paths are sorted in byte order. The first is `first`, and + every other path is a `dupe`. +- Groups are ordered by size, largest first, then by `first` path. The output is + the same for the same database contents. -`report` runs in two phases, each with its own progress display on stderr (see -"Progress"): `open` opens the database; `read` counts the records, then reads -them in signature order, largest size first. That is the order of the -`files_signature` index read backwards, so SQLite sorts nothing and returns the -first record at once. The records sharing a signature come together, and so do -the records of one size, so `report` holds only the duplicate groups of the size -it is reading, and writes them, ordered by `first` path, once it reaches a -smaller size. Its memory holds at most the duplicate paths of one size at a -time, never every record. +`report` has two phases, each with its own progress display on stderr (see +"Progress"). `open` opens the database. `read` gets the number of records from +the `totals` table (see "Schema"), then reads them in signature order, largest +size first. The database keeps an index named `files_signature`, a sorted list +of the records by signature (see "Schema"). Reading it backwards gives that +order, so SQLite sorts nothing and returns the first record at once. Records +with one signature come together, and so do the records of one size. So `report` +holds only the duplicate groups of the size it is reading. Once it reaches a +smaller size, it writes them, ordered by `first` path. #### Report output format -TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a -group of N): +The report is TSV (tab-separated values) on stdout. It holds a header line, then +one row per duplicate file. A group of N paths gives N-1 rows. ``` first dupe size @@ -757,99 +691,152 @@ first dupe size /srv/a/big.iso /srv/c/big-copy2.iso 4294967296 ``` -Paths are raw bytes and may hold any byte except NUL, so the path columns -(`first` and `dupe`) are escaped to keep every row one line of tab-separated -fields: a backslash is written as `\\`, a tab as `\t`, a newline as `\n`, and a -carriage return as `\r`. Every other byte is written unchanged, including bytes -that are not valid UTF-8. Undoing those four escapes gives back the stored path. -Grouping and ordering use the stored path, not the escaped one. The warnings -`scan` prints on stderr are escaped the same way, so each warning is one line. +Paths are raw bytes and may hold any byte except NUL, the zero byte. The path +columns are escaped, so that every row stays one line of tab-separated fields. +Four bytes are written as escapes: -Summary to stderr: records read, number of duplicate groups, number of dupe -files, and total reclaimable bytes (sum of `size` over all dupe rows) in human -units. +- a backslash as `\\`; +- a tab as `\t`; +- a newline as `\n`; +- a carriage return as `\r`. + +Every other byte is written unchanged, including bytes that are not valid UTF-8, +the usual text encoding. Undoing the four escapes gives back the stored path. +Grouping and ordering use the stored path. The warnings about skipped paths that +`scan` prints are escaped the same way, so each stays one line. + +The summary line on stderr gives: + +- the records read; +- the duplicate groups; +- the dupe files, which is the number of rows; +- the reclaimable bytes, the sum of `size` over all rows. + +``` +report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable +``` ### `trees` mode -`trees` reads the same database as `report` (no positional arguments) and -reports **entire duplicate directory trees**: directories under which the exact -same set of relative paths exists with the exact same file signatures. - -**`trees` must never touch the filesystem being analyzed** — the same rule as -`report`. The directory hierarchy is reconstructed purely from the paths in the -records, split on `/`. +`trees` reads the same database as `report` and takes no positional arguments. +It reports whole duplicate directory trees. Two directories are duplicate trees +when they hold the same relative paths, and each path holds a file with the same +signature in both. Like `report`, it never touches the scanned filesystem. It +rebuilds the directory tree from the paths in the records, split on `/`. Definitions: -- A file's **signature** is `(size, head, tail, content)` — mtime is - informational and excluded. A record without a `content` hash has unknown - content: its signature is treated as unique to that file, so a tree containing - such a file never compares equal to any other tree. -- A directory's **digest** is a SHA-256 Merkle digest computed bottom-up: - serialize the directory's child entries — for a file child, its name and - signature; for a subdirectory child, its name and that subdirectory's digest — - sort the serialized entries byte-lexicographically, and hash the - concatenation. Names are part of the digest: two trees whose files differ only - in name are _not_ duplicates. +- A file's **signature** is `(size, head, tail, content)`, as for `report`. The + mtime is not part of it. +- A directory's **digest** is a SHA-256 hash computed from the bottom up. Each + child contributes one entry. A file's entry is its name and signature. A + subdirectory's entry is its name and digest. The entries are sorted in byte + order and hashed together. Names are part of the digest, so two trees whose + files differ only in name are not duplicates. - Two directories are **duplicate trees** when their digests are equal. Equal - digests imply equal recursive file count and equal total byte size. - -Known limitation (accepted): hard-linked paths are reported as duplicates by -`report` and count toward duplicate trees — their content is genuinely identical -— even though they share storage, so removing one reclaims no space. Inode -identity is used during the scan to avoid redundant reads but is not persisted -in the database. - -Known limitation (accepted): only regular files that appear in the database -define a tree. Empty directories are invisible, and a file skipped during the -scan (e.g. permission error) in one copy but not the other will make -otherwise-identical trees compare as different. + digests imply the same number of files and the same total size. Processing: -- Build the hierarchy, compute every directory's digest, and group directories - by digest. Every group with two or more directories is a duplicate-tree group. -- **Report only maximal trees.** A group is suppressed when its members' parents - are pairwise distinct directories that all share a single digest — such a - group is wholly implied by its parents' (or a further ancestor's) group. - Groups containing sibling directories, or members whose parents differ, are - always reported. -- Within each group, sort paths lexicographically (byte order); the first path - is `first`, every other path is a `dupe`. -- Order groups by total tree size descending, tie-broken by `first` path - ascending. Output must be fully deterministic for a given input. +- `trees` builds the directory tree and computes every directory's digest. It + then groups the directories by digest. Every group with two or more + directories is a duplicate-tree group. +- `trees` leaves out a group that a group of its members' parents already + implies. A group is left out when its members' parents are distinct + directories that all have the same digest. The parents' group, or an + ancestor's, already implies it. A group is always reported when it holds + sibling directories, or when its members' parents have different digests. +- Within a group, paths are sorted in byte order. The first is `first`, and + every other path is a `dupe`. +- Groups are ordered by total tree size, largest first, then by `first` path. + The output is the same for the same database contents. -`trees` runs in four phases, each with its own progress display on stderr (see -"Progress"): `open` opens the database; `read` counts the records, then reads -them in path order, building the hierarchy and computing the digests as it goes; -`group` groups the directories by digest and picks the maximal groups; `write` -writes the groups to stdout. +Two limitations are accepted: + +- Hard-linked paths count toward duplicate trees. Their content is identical. + They share storage, so removing one frees nothing. The scan uses the inode to + avoid reading a file twice, and does not store it. +- Only the regular files in the database define a tree. Empty directories are + invisible. When the scan skips a file in one copy but not in the other, the + two trees differ. + +`trees` has four phases, each with its own progress display on stderr (see +"Progress"). `open` opens the database. `read` gets the number of records from +the `totals` table, then reads them in path order. It builds the directory tree +and computes the digests as it goes. `group` groups the directories by digest +and picks the groups to report. `write` writes them to stdout. #### Trees output format -TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a -group of N). `files` is the recursive regular-file count of one copy of the -tree; `size` is the recursive total byte size of one copy: +The report is TSV on stdout, with a header line and then one row per duplicate +tree. A group of N trees gives N-1 rows. After the `first` and `dupe` paths come +three numbers: + +- `files`, the number of regular files in one copy of the tree; +- `size`, their total size in bytes; +- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size` + added up. ``` -first dupe files size -/srv/a/project /srv/backup/project 3417 104857600 +first dupe files size disk_size +/srv/a/project /srv/backup/project 3417 104857600 61865984 ``` -The `first` and `dupe` paths are escaped as described under "Report output -format". The root directory's path is `/`. +A file's `disk_size` is the space its blocks take on disk. It can be more than +the file's size, because space is allocated in whole blocks. It is less than the +file's size on a compressed or sparse file. A sparse file has gaps that take no +space on disk. So two copies of one tree can differ in `disk_size`. When a file +in one copy is a hard link to one in the other, both copies count its blocks in +full. Removing it from one copy then frees nothing, as the limitation under +"`trees` mode" says. -Summary to stderr: records read, number of duplicate-tree groups, number of dupe -trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human -units. +The paths are escaped as described under "Report output format". The root +directory's path is `/`. + +The summary line on stderr gives: + +- the records read; +- the duplicate-tree groups; +- the dupe trees, which is the number of rows; +- the reclaimable bytes, the sum of `size` over all rows. + +``` +trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable +``` + +### `dwim` mode + +`dwim`, short for "do what I mean", scans a new machine and reports its +duplicate trees in one command. It takes no positional arguments and no flags of +its own. Like `scan`, it should run as root, so that it can read every file. + +`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default +excludes and crosses into other filesystems. `--workers` keeps its default. The +database, its creation and the scan lock are those of `scan`. Then `dwim` runs +what `sfdupes trees` runs. The largest trees come first. + +The report goes to stdout, exactly as `trees` prints it. On stderr comes what +`scan` prints, then what `trees` prints, then one last line: + +``` +dwim: run "sfdupes trees" to print this report again +``` + +The database keeps the scan, so `sfdupes trees` prints the same report later, +even after the terminal is gone. + +`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan"). +`dwim` then prints no report and exits 1. A fatal scan error also ends it before +the report, with the same message and exit code as `scan`. Warnings for skipped +files do not stop it. `dwim` stops catching signals when the scan ends, so a +signal during the report ends it at once, as it ends `trees`. ### `info` mode `info` takes no positional arguments and prints the database's details on -stdout, one per line: a name, a tab, and the value. It opens the database the -way `report` does, read-only and without the scan lock, so it runs during a scan -and needs only read access (see "Database"). Like `report`, it never touches the -scanned filesystem. +stdout, one per line. Each line holds a name and its value, separated by a tab. +`info` opens the database read-only, as `report` does, so it runs during a scan +and needs only read access. It never touches the scanned filesystem. ``` database /var/lib/sfdupes/db.sqlite (default) @@ -858,325 +845,536 @@ database file size 2147483648 (2.0 GiB) modified 2026-10-16T12:02:50Z schema version 1 records 12345678 -records with head and tail hashes 1234567 -records with content hash 123456 size of recorded files 150000000000000 (136.4 TiB) ``` - `database` is the path and where it came from, as on the line every subcommand prints first on stderr (see "Database"). -- `database file size` and `-wal file size` are in bytes, then in human units. - The `-wal` line is there only when a `-wal` file is beside the database, as - while a scan runs. -- `modified` is the time the database file was last changed, in UTC. While the - database is in WAL mode, the newest changes may be in the `-wal` file alone. -- `schema version` is the database's `PRAGMA user_version`. A database with - another version than this build's is refused, as by `report`. -- `records` is the number of records. `records with head and tail hashes` and - `records with content hash` count the records that have those hashes (see - "Database"), and `size of recorded files` adds up the sizes of all records, in - bytes, then in human units. +- `database file size` and `-wal file size` are in bytes, then rounded to a unit + such as MiB. The `-wal` line appears only while a `-wal` file is beside the + database, as during a scan. +- `modified` is when the database file was last changed, in UTC. While a scan + runs, the newest changes may be in the `-wal` file alone (see "Concurrent + reads"). +- `schema version` is the version of the database's layout, read from + `PRAGMA user_version` (see "Schema"). A database with another version than + this build's is refused, as by `report`. +- `records` is the number of records. `size of recorded files` adds up the sizes + of all recorded files, in bytes, then rounded to a unit such as TiB. -`info` runs in two phases, each with its own progress display on stderr (see -"Progress"): `open` opens the database; `count` reads the schema version and -counts the records with one query, a single pass over the table, both in one -read-only transaction, so the numbers agree with each other while a scan is -committing. `info` writes its lines once the `count` display has ended, so on a -terminal they come after the displays. +`info` has two phases, each with its own progress display on stderr (see +"Progress"). `open` opens the database. `count` reads the schema version and the +row of `totals` (see "Schema") in one transaction. A transaction is a group of +reads or writes that SQLite carries out as one step, against the database as it +stands at one moment. So both describe the database at one moment, even while a +scan is writing. `info` writes its lines after the `count` display has ended. + +### Database + +All five subcommands work on one SQLite database file. + +#### Location + +The database is the file named by `SFDUPES_DATABASE`, when that is set and not +empty, and otherwise `/var/lib/sfdupes/db.sqlite`. + +Each subcommand prints one line on stderr naming the database it uses, before +anything else and whether or not stderr is a terminal. The line gives the path +and where it came from: + +``` +scan: new database /var/lib/sfdupes/db.sqlite (default) +trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE) +``` + +`dwim` prints the `scan` line first, and the `trees` line after the scan. + +A relative path is printed with the working directory in front of it. The +working directory is printed as `pwd -P` prints it, with its symlinks resolved. +The rest of the path is printed as set, so a `..` in it stays. The line names +the file the setting names. After a `cd` through a symlink, a `..` counts from +the directory the symlink leads to, as it does for any other program. The +printed path shows that directory. The path is escaped like the paths in a +report. `scan` says `new database` when no file is at that path as it starts. +`report`, `trees` and `info` print the line before they open the database, so +the error for a missing database comes after it. + +#### Creation and access + +- `scan` creates the database and its parent directory on first use. +- `report`, `trees` and `info` need an existing database. A missing database is + a fatal error, exit 1. The error tells you to run `scan` first. +- `report`, `trees` and `info` open the database read-only. They need read + access to the database file and no write access to its directory. While the + database has `-wal` and `-shm` files beside it (see "Concurrent reads"), they + read those too. SQLite creates them with the database file's permissions. + +#### The scan lock + +Only one `scan` at a time can hold the lock. + +- For its whole run, `scan` holds a lock on a lock file beside the database. The + lock is taken with the `flock(2)` system call, a request a program makes to + the kernel, and only one process can hold it. The lock file's name is the + database's path, with symlinks resolved, plus `.lock`. By default that is + `/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same + lock file. +- `scan` takes the lock before it walks the filesystem or opens the database. + Just before that, it creates the database file, empty, if it is missing. So + the lock file is named the same way before the first scan as after. A symlink + at the database path is followed, and its target is created. +- A second `scan` that finds the lock held does not wait. It fails at once with + an error naming the lock file, and exits 1. It has walked nothing and has not + opened the database. The running scan carries on. +- The lock file is created on first use and never deleted. Only its owner may + open it. A leftover file blocks nothing. The lock ends with the process that + holds it, however that process ends. Deleting the file while a scan runs would + let a second scan start. +- `report`, `trees` and `info` never take the lock, so they run during a scan. + +#### Concurrent reads + +- While `scan` runs, the database is in write-ahead log (WAL) mode. In WAL mode + SQLite appends changes to a `-wal` file beside the database, and keeps a + lookup table for that file in a `-shm` file. Readers do not block the writer, + so running a report during a scan is safe. +- Every subcommand opens the database with a busy timeout of 10 seconds. That is + how long it waits for a database another process has locked before it gives + up. +- The filesystem is authoritative, and the database follows it. Hashed records + are written 10,000 at a time while the scan runs, so a report may see a scan's + changes partly applied. A scan that dies partway leaves a valid database + holding everything written so far, and the next scan carries on from there. +- When `scan` closes the database, it switches it back to SQLite's default mode, + called rollback-journal mode. In that mode the database file alone holds + everything. Between scans a reader therefore needs no `-wal` or `-shm` file. + Each switch needs the database to itself. A `scan` that starts while a report + is still reading waits for it up to the busy timeout, then fails. A `scan` + that ends while a report has the database open prints a warning and leaves the + database in WAL mode until the next scan. `report` writes its rows while it + reads. So it is still reading while a pager or a stalled pipe holds up its + output. A pager, such as `less`, shows output one screen at a time and waits + for a key press. + +#### What `scan` writes, and when + +- As it starts, before the walk, `scan` creates whichever of these are missing: + - the database's directory; + - an empty database file; + - the `.lock` file. +- Then, before the walk as well, it writes: + - the `-wal` and `-shm` files, which stay beside the database while the scan + runs; + - in a new database, the tables, the index, the triggers and the schema + version described under "Schema". +- During the `load` and `walk` phases, no records are written. A first scan's + database holds no records until the hash phase, however long its walk runs. +- During the `hash` phase, records are written 10,000 at a time. +- During the `update` phase, the rest of the hashed records are written, and so + is the new `disk_size` of each unchanged file whose size on disk changed. Then + the deletions are made. +- As `scan` closes the database, the `-wal` and `-shm` files go away, unless a + report still has the database open. + +#### Schema + +The schema is the database's layout: its tables, its index and its triggers. A +trigger is a statement SQLite runs by itself when a table changes. +`PRAGMA user_version` holds the schema version, currently 1. An empty database +is at version 0, and `scan` sets it up. Any other version is a fatal error. +`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal +error too. `scan` creates the tables, the index and the triggers and sets the +version in one transaction (see "`info` mode"). So they all land together or not +at all. A first scan stopped while doing so leaves an empty database, which the +next scan sets up. A database at version 0 that already has a `files` table was +not made by `sfdupes`. Every subcommand refuses it, with an error telling you to +remove the file and rescan. + +```sql +CREATE TABLE files ( + path BLOB PRIMARY KEY, -- absolute path, raw bytes + size INTEGER NOT NULL, -- bytes + disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512) + mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime + mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second + head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB + tail BLOB NOT NULL, -- raw 32-byte SHA-256 of the last 64 KiB + content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples +) WITHOUT ROWID; +CREATE INDEX files_signature ON files (size, head, tail, content); +CREATE TABLE totals ( + records INTEGER NOT NULL, -- number of records in files + size INTEGER NOT NULL -- sizes of all records in files added up +); +INSERT INTO totals (records, size) VALUES (0, 0); +CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN + UPDATE totals SET records = records + 1, size = size + NEW.size; +END; +CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN + UPDATE totals SET records = records - 1, size = size - OLD.size; +END; +CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN + UPDATE totals SET size = size + NEW.size - OLD.size; +END; +``` + +- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not + necessarily UTF-8 text. +- `disk_size` is the space the file's blocks take on disk. `lstat` is the system + call that stats a file without following a symlink. Its `st_blocks` field + counts that space in units of 512 bytes. `disk_size` is less than `size` on a + compressed or sparse file (see "Trees output format"). +- `mtime` holds the whole seconds of the file's mtime, rounded down, and + `mtime_nsec` the nanoseconds past that second. Split this way, they hold any + mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit + count of nanoseconds cannot hold those. They serve change detection only and + are not part of the signature. +- `head`, `tail` and `content` each hold the raw 32 bytes of a SHA-256 (see + "Duplicate detection"). `report` and `trees` fail on a record whose hash is + not 32 bytes. +- `files_signature` lets `report` read the records in signature order without + sorting the table. +- `totals` holds one row: the number of records in `files` and the sum of their + sizes. Three triggers on `files` keep that row current. Each runs in the + transaction of the change to `files` that fires it, so the row and the records + always agree. `info` reads both numbers from this row. The `load` phase of + `scan` and the `read` phase of `report` and `trees` read the number of records + from it as their total. Counting the records in `files` would read every + record. That takes minutes on a database of tens of millions. The triggers + make each insert, each delete and each change of a record's `size` slower, so + a first scan of many small files takes about two fifths longer. On large + files, reading them takes most of the time, so the difference is small. ### Progress -Use the progress-bar library for all progress of `scan`, `report`, `trees` and -`info`; rendering in the style of `pv` is the model. All progress goes to -stderr. +Each phase of every subcommand gets a display of its own on stderr from the +moment it starts. The display names the phase and shows its progress. The parts +of its line are listed below. -`scan`, `report`, `trees` and `info` must always show that they are working and -not hung. Each phase gets its own display from the moment it starts, naming the -phase and its place among the command's phases and showing its progress, its -elapsed time and its estimated time remaining (ETA). +The phases, and what each display counts: -A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`, -`[4/5] update`, `[5/5] content`): - -- `load` counts the records first, then reads them against that count. While the - count runs, the display says it is counting the records. -- `walk` has no exact total. Its estimate is the number of records the last scan - left under the operands, known once `load` is done and shown with a `~`. Once - the walk passes the estimate, or when there are no such records, as on a first - scan, the ETA is unknown. -- `hash` and `update` have exact totals. Only files that actually need hashing - appear in the hash total, so its ETA is meaningful. -- `content` first counts the records whose files it checks with `lstat`, which - SQLite does by adding up the number of records with each matching size, - `head`, and `tail`, without fetching the records. It then says it is searching - the database until SQLite returns the first record; the checks, and their rate - and ETA, start there. The files to read then get a display of their own, with - their count as its total, and so does the writing of the last of their records - to the database, with the count of those records as its total. - -`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`, -`[4/4] write`, see "`trees` mode"): - -- `open` has no total. While it opens the database, waiting on a busy one - included, the display says it is opening the database. -- `read` counts the records first, then reads them against that count. While the - count runs, the display says it is counting the records. -- `group` has the directories `read` found as its total, and `write` the - duplicate-tree groups. - -`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"): - -- `open` has no total. While it opens the database, waiting on a busy one - included, the display says it is opening the database. -- `read` counts the records first, then reads them against that count, writing - the rows to stdout as it goes. While the count runs, the display says it is - counting the records. - -`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"): - -- `open` has no total. While it opens the database, waiting on a busy one - included, the display says it is opening the database. -- `count` has no total either, because one query counts everything: the display - says it is counting the records until the query returns, and its ETA is - unknown. +- `scan` has four phases: + - `[1/4] load` gets the number of records from the `totals` table first, + then reads them against that number. Until it has the number, the display + says `counting the records`. + - `[2/4] walk` has no exact total. Its estimate is the number of records + under the operands, shown with a `~`. A scan with no records under its + operands, such as a first scan, has no estimate. The time remaining is + `unknown` when there is no estimate, and once the walk passes the + estimate. + - `[3/4] hash` and `[4/4] update` have exact totals. Only the files that + need hashing count toward the hash total, so its time remaining is + meaningful. +- `report` has two phases: + - `[1/2] open` has no total. The display says `opening the database`, also + while it waits on a busy one. + - `[2/2] read` gets the number of records from the `totals` table first, + then reads them against that number. Until it has the number, the display + says `counting the records`. It writes rows to stdout as it goes. +- `trees` has four phases: + - `[1/4] open` and `[2/4] read` are as for `report`, except that `read` + writes nothing. + - `[3/4] group` has the directories that `read` found as its total. + - `[4/4] write` has the duplicate-tree groups as its total. +- `info` has two phases: + - `[1/2] open` is as for `report`. + - `[2/2] count` has no total either, because it reads one row. The display + says `counting the records` until it has the row. Each display is one line holding: -- the phase and its place among the phases -- a `[m/n] x%` display (items processed / total items, percent); before a phase - has a total, the items processed, or what the phase is waiting for -- current rate (items/s), naming the items the phase counts: files, records, - directories or groups. It covers the items completed over the last 5 seconds, - or, when stderr is not a terminal, since the line before, so it falls to 0 - while nothing completes, and a burst of items after a wait shows as a burst - rather than as a steady rate. Once no item has completed for 10 seconds, the - word `stalled` and where the phase waits take the rate's place (see below). -- elapsed time -- estimated time remaining, or `unknown`. It comes from the phase's average rate - since it started counting, so that it does not swing with each burst. +- the phase and its place among the phases; +- the progress, written `[m/n] x%` for `m` items done of a total `n`, which is + `x` percent of it. A phase that has no total shows the items done instead, as + `1234 files`. A phase waiting for its total shows what it waits for, as + `counting the records`, and no rate; +- the current rate, in the items the phase counts, such as files or records. It + covers the items completed over the last 5 seconds, or, when stderr is not a + terminal, since the line before. It falls to 0 while nothing completes. A + burst of items after a wait shows as a high rate for the next 5 seconds, then + the rate falls again; +- the elapsed time, labelled `elapsed`; +- the time remaining, labelled `remain`, as a duration such as `12m9s` or as + `unknown`. It comes from the phase's average rate since it started counting, + so it does not swing with each burst. -Example shape (exact layout is flexible, content is not): +Example lines, as printed when stderr is not a terminal: ``` -[3/5] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54 -[2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown -[3/5] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s +[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s +[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown +[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s ``` -When no item has completed for 10 seconds, as while one large file is read, a -disk is slow or the database is busy, the line says `stalled` and where the -phase waits in place of the rate, which would read 0: `disk` for files, `db` for -the database, `out` for stdout, and two of them, as `disk/db`, for a phase that -may wait on either. `walk` says `disk`. `hash` and the content phase's reads say -`disk/db`, as they read files and write their records to the database between -them, and so do the content phase's checks, which `lstat` each record's file as -the database search returns it. `load`, `update`, the content phase's last write -and the `read` phase of `trees` say `db`. `trees`' `write` says `out`, and its -`group`, which works in memory, says `cpu`. The `read` phase of `report` writes -the report while it reads the records, so it says `db/out`. A phase still -waiting for its total already says what it waits for, and says nothing more. +When no item has completed for 10 seconds, the line says `stalled` and where the +phase waits, in place of the rate. That happens while one large file is read, or +while a disk or the database is slow. Each phase names the places it may wait +in, and a phase that may wait in two places names both, as `disk/db`: -Additional requirements: +- `walk` says `disk`, because it reads directories and stats files. +- `hash` says `disk/db`, because it reads files and writes their records to the + database between them. +- `load`, `update` and the `read` phase of `trees` say `db`. +- The `read` phase of `report` says `db/out`, because it writes the report to + stdout while it reads. +- `group` of `trees` says `cpu`, for work in memory, and its `write` says `out`. +- A phase still waiting for its total says what it waits for, and nothing more. -- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`, - the `open` phase of `report` and `trees`, and both phases of `info`, once it - has counted what it will do) is drawn as its line followed by the progress-bar - library's bar, 10 columns wide, which shows the percent in place of the line's - own. Any other phase, and a phase whose bar has no room, is drawn as the - library's spinner followed by its line. Each display is drawn again several - times a second, so its elapsed time keeps moving while a phase waits for its - next item. -- On a terminal no line reaches the terminal's last column, however narrow the - terminal is. A line too wide for the terminal first loses its bar, which gives - way to the spinner, then its rate, the words of a stalled phase, or what the - phase is waiting for. The phase and its place, the progress, the elapsed time - and the ETA are kept whole. On an 80-column terminal, at counts like those of - the stalled line in the example above, a stalled line keeps its words whenever - the elapsed time and the ETA are both in minutes, except in the content - phase's checks and reads. Their lines are the widest, and at most times in - minutes they lose the words. Where the words and the ETA do not both fit, as - at millions of files and times in hours, the words give way. Only when those - four alone are too wide, as on a terminal narrower than about 70 columns hours - into a scan of millions of files, is the line cut short at the terminal's - edge, so the end of the ETA is lost first. A terminal made narrower while a - phase runs gets the same: the next draw erases the line within the new width - and fits the line to it, and the display goes on redrawing in place. -- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI - redraws: print the line the moment each phase starts, every 5 seconds while it - runs (whether or not an item has completed, and never more often), and once - more as the phase ends. -- Progress must be non-blocking with respect to the worker pool: the main - goroutine counts the items, and each display draws them from a goroutine of - its own. -- A warning printed during a phase always lands on a line of its own, never - inside the progress display. So does every line written to stdout when stdout - and stderr are the same file (the same device and inode): one terminal, as in - a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are - sent to. When they are not, stdout is written without waiting on the display, - and the display keeps being drawn while a slow reader of stdout holds up a - write. -- A phase that stops short of its total, as an interrupted one does, is left - showing the count it stopped at rather than filled up. +On a terminal: + +- Every display starts with a spinner, a turning character, followed by its + line. The display is drawn again several times a second. So the spinner turns + and the elapsed time keeps moving while a phase waits for its next item. +- A phase with an exact total ends its line with its percent and a bar 10 + columns wide, at the terminal's right edge. The percent after the count is + left out then. A phase with nothing to do shows no bar. Every phase has an + exact total once it has counted its work, except these: + - `scan`'s `walk`; + - the `open` phase of `report` and `trees`; + - both phases of `info`. + + On a terminal 100 columns wide: + + ``` + ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ | + ``` + +- No line reaches the terminal's last column. A line too wide for the terminal + first leaves out its rate, the words of a stalled phase, or what the phase is + waiting for. It leaves out its bar next. It is cut short at the edge only when + the rest does not fit on its own. The rest is: + - the phase and its place; + - the progress, when the phase shows one; + - the elapsed time; + - the time remaining. + + On a terminal 80 columns wide, no line is cut short, even hours into a scan + of millions of files. A terminal made narrower during a phase is fitted at + the next draw. + +- A warning printed during a phase lands on a line of its own, never inside the + display. So does every line written to stdout when stdout and stderr are the + same file. That is one terminal, or one pipe or file that both are sent to. + When they are not the same file, stdout is written without waiting on the + display. The display then keeps being drawn while a slow reader holds up + stdout. + +When stderr is not a terminal, such as a pipe or a file, there are no redraws. +Each phase prints its line: + +- as it starts; +- every 5 seconds while it runs, whether or not an item has completed; +- once more as it ends. + +When a phase stops short of its total, as an interrupted one does, it keeps +showing the count it stopped at. ### Error handling and exit codes -- `0`: success, even if individual files were skipped with warnings. -- `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is - already running against the same database, the database cannot be - created/opened/read/written, a missing database for `report`/`trees`/`info`, - stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see - below). -- `2`: usage error (including `scan` with no `PATH` operand, `scan` with - `--workers` below 1, and `report`/`trees`/`info` with any positional - argument). +- `0`: success, even when files were skipped with warnings. +- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim` + stopped by either during its scan. A fatal error is one of these: + - an operand that does not exist, or whose real path `scan` cannot find; + - a list of mounted filesystems that `scan` cannot read; + - another `scan` holding the lock (see "The scan lock"); + - a database, its directory or its lock file that cannot be created or + opened; + - a read or write of the database that fails; + - a relative database path when the working directory cannot be read; + - a database at a schema version other than 0 or 1; + - a database at version 0 read by `report`, `trees` or `info`; + - a database at version 0 that already has a `files` table (see "Schema"); + - a record with a hash that is not 32 bytes, read by `report` or `trees`; + - a missing database for `report`, `trees` or `info`; + - a failed write to stdout. +- `2`: a usage error, which is one of these: + - a missing or unknown subcommand; + - an unknown flag; + - a flag without its value, or with a value that cannot be parsed, such as + `--workers abc`; + - `scan` with no operand; + - `--workers` below 1; + - `report`, `trees`, `info` or `dwim` with a positional argument. -A stdout write failure, such as a full disk, is reported in one line on stderr -and exits 1. Two cases never reach sfdupes as a failed write: +A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to +stdout, such as a full disk, is reported the same way. Two cases never reach +`sfdupes` as a failed write: -- When the reader of a stdout pipe exits early, as in `sfdupes report | head`, - the next write ends sfdupes with `SIGPIPE`, quietly and without a summary, the - way `cat` or `sort` end. The shell reports the signal (status 141 in most - shells), not exit 1. -- When stdout is closed outright (`sfdupes report >&-`), the Go runtime opens - `/dev/null` in its place before sfdupes starts, so the output is discarded and - the run succeeds, as with `> /dev/null`. - -`scan` stops cleanly on `SIGINT` (Ctrl-C) or `SIGTERM`. Its workers stop taking -work, each finishing at most the directory listing or file it is reading; the -progress display is finished; and the records it has hashed but not yet -committed are committed, so the next scan does not hash them again. Apart from -that commit it starts no further writes or deletions: records are deleted only -after a complete walk, so those under paths an interrupted walk never reached -are kept. The database is closed and the lock released as on any other exit, the -line `scan: interrupted after N files` goes to stderr, N being the number of -files the walk reached, and the exit code is 1. The next scan skips the records -already written and converges as usual. - -After the first signal `scan` stops catching them, so a second one ends it at -once, as an uncaught signal does: the records not yet committed are lost, and -the database is left valid, as when any scan dies (see "Database"). A `SIGINT` -that `scan` inherits as ignored, as a script's background job does, stays -ignored. +- The reader of a stdout pipe may exit early, as `head` does in + `sfdupes report | head`. The next write then ends `sfdupes` with `SIGPIPE`, + the signal for a write to a closed pipe. `sfdupes` ends quietly and without a + summary, as `cat` or `sort` would. The shell reports the signal, status 141 in + most shells, not exit 1. +- When stdout is closed outright, as in `sfdupes report >&-`, the Go runtime + opens `/dev/null` in its place before `sfdupes` starts. The output is + discarded, and the run succeeds, as with `> /dev/null`. ## Entrypoints -This repository adheres to the +This repository follows the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) -standard: the normalized executables in `script/` are the entrypoints for the -development workflow, and the `Makefile` targets are thin shims that call them. -Every script is POSIX `sh`, resolves the repository root itself so it can be run -from any working directory, and may be invoked directly. The provided -entrypoints are: +standard. The executables in `script/` are the entrypoints of the development +workflow, and the `Makefile` targets call them, except `build` and `clean` (see +"Build"). Every script runs under plain `sh`, the standard Unix shell. It finds +the repository root itself, so it can be run from any directory. -- `script/bootstrap` — install everything needed to build and develop this - repository, idempotently, assuming nothing is present. `git`, `make`, and `go` - come from the first of nix, apt, brew, or apk found on the host, and are - presence-checked only. An installed node is used when its major version is 22; - otherwise node 22.17.0 is installed through nvm, which comes from a release - archive whose sha256 the script checks, and yarn 1.22.22 is activated under it - through corepack. With an installed node 22, a yarn already on `PATH` is used - as it is; otherwise yarn 1.22.22 is activated through corepack, or installed - with `npm` when there is no corepack. `yarn install --frozen-lockfile` then - installs the prettier that `package.json` and `yarn.lock` pin. `golangci-lint` - is never installed: it runs in Docker (see `script/lint`). `docker` is not - installed either; testing, linting and the image build need it. Ends with - `go mod download`. -- `script/setup` — make a fresh clone ready for development: runs +- `script/bootstrap` installs what building and developing need. It can be run + again at any time, and installed tools are kept. `git`, `make` and `go` are + checked for presence only. A missing one is installed with the first of these + package managers found on the host: + - `nix`; + - `apt`; + - `brew`; + - `apk`. + + Then come `node` and `yarn`. `prettier`, the Markdown formatter, is a `node` + program. `yarn` is the package manager that installs it: + - An installed `node` is used when its major version is 22. A `yarn` already + on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is + activated through `corepack`, the `yarn` installer that ships with `node`. + When there is no `corepack` either, the script installs `yarn` with `npm`, + the package manager that ships with `node`. + - Otherwise the script installs `node` 22.17.0 through `nvm`, a `node` + version manager, and activates `yarn` 1.22.22 under that `node` through + `corepack`. When `nvm` is missing, the script first downloads it and + checks the archive's SHA-256. + + `yarn install --frozen-lockfile` then installs the `prettier` that + `package.json` and `yarn.lock` pin. The linter is never installed. It runs + in Docker (see `script/lint`). `docker` is not installed either. + `make check` and `make docker` both need it. The script ends with + `go mod download`. + +- `script/setup` makes a fresh clone ready for development. It runs `script/bootstrap`, then `script/install-precommit`. -- `script/projectname` — print this project's name (`sfdupes`). Scripts that - need the name call it, so they stay identical across repositories. -- `script/test` — run the test suite under the race detector, with a 90-second - timeout and coverage enabled, rerunning verbosely on failure so the logs show - which test failed. It builds the `Dockerfile`'s `test` phase alone and writes - no image. The race detector needs cgo and a C compiler, which the build never - uses, so the phase starts from a digest-pinned Debian `golang` image, which - has `gcc`. The tests run as `nobody`, because several of them make a file - unreadable and root reads it anyway. -- `script/lint` — run the linter. It builds the `Dockerfile`'s `lint` phase - alone and writes no image: in the digest-pinned `golangci/golangci-lint` - image, the gofmt check, `golangci-lint config verify` and `golangci-lint run` - run as build steps, so a successful build is a clean lint. The linter is never - run on the host, which makes a working `docker` the one prerequisite for - linting. -- `script/fmt` — format in place: the Go sources with `gofmt -s -w`, and every - Markdown file with prettier, at the settings in `.prettierrc` (4-space - indents, prose wrapped at 80 columns). Both run on the host, prettier through - yarn at the version `yarn.lock` pins. When the node on `PATH` is not node 22, - this loads the node 22.17.0 that `script/bootstrap` installed through nvm. -- `script/fmt-check` — the read-only counterpart of `script/fmt`: prints any - unformatted file and exits non-zero instead of writing. gofmt and prettier +- `script/projectname` prints `sfdupes`. Scripts that need the name call it, so + they stay identical across repositories. +- `script/test` runs the test suite under the race detector, Go's check for + unsynchronized access to memory from several threads. The suite runs with a + 90-second timeout, and its coverage of the code is measured. On failure it + reruns the suite verbosely, so that the log names the failing test. It builds + only the `test` stage of the `Dockerfile` and writes no image. A stage is one + named part of the `Dockerfile`, which starts from an image of its own and can + be built alone. The race detector needs cgo, Go's bridge to C code, and a C + compiler. The stage therefore starts from a Debian `golang` image, which has + the C compiler `gcc`. The image is pinned by its digest, its content hash. The + tests run as the user `nobody`, because several of them make a file + unreadable, and root reads it anyway. +- `script/lint` runs the linter, `golangci-lint`, which checks the Go sources + for common mistakes. It builds only the `lint` stage of the `Dockerfile` and + writes no image. The stage starts from the `golangci/golangci-lint` image, + pinned by its digest, and runs these checks as build steps: + - the `gofmt` check, which fails when `gofmt`, Go's own formatter, would + change a Go source; + - `golangci-lint config verify`, which checks the linter's settings file + `.golangci.yml`; + - `golangci-lint run`, which lints the Go sources. + + A successful build is a clean lint. The linter never runs on the host, so a + working `docker` is the one thing linting needs. + +- `script/fmt` formats the Go sources with `gofmt -s -w` and every Markdown file + with `prettier`, in place. `prettier` indents by 4 spaces and wraps prose, as + `.prettierrc` sets and as the script also passes on the command line. It wraps + at 80 columns, its default width. Both run on the host. `prettier` runs + through `yarn` at the version that `yarn.lock` pins. When the `node` on `PATH` + is not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap` + installed. +- `script/fmt-check` is the read-only form of `script/fmt`. It names any + unformatted file and exits non-zero instead of writing. `gofmt` and `prettier` both run every time, and each names itself when it fails. The `Dockerfile`'s - `lint` phase runs the same gofmt check. -- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in - that order. Modifies nothing. The first two need `docker`, the third what - `script/bootstrap` installs. -- `script/docker` — build the Docker image, tagged with the name from - `script/projectname`, passing the output of - `git describe --tags --always --dirty` (or `unknown` when that is empty) as - the `VERSION` build argument. The `Dockerfile`'s build stage depends on its - `lint` and `test` phases, so this also runs the tests and the linter. -- `script/cibuild` — run `script/bootstrap`, then `script/check`, then build the - image as `script/docker` does. This is what the Gitea workflow runs on push; a - successful run means every check passed. The tests and the linter run twice: - once in `script/check`, and again as stages of the image build. -- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a - no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then - `script/check`. -- `script/install-precommit` — install the git pre-commit hook, as - `.git/hooks/pre-commit`, that runs `script/precommit`. + `lint` stage runs the same `gofmt` check. +- `script/check` runs three scripts in order, and changes nothing: + - `script/test` and `script/lint`, which need `docker`; + - `script/fmt-check`, which needs what `script/bootstrap` installs. +- `script/docker` builds the Docker image, tagged with the name from + `script/projectname`. It passes the output of + `git describe --tags --always --dirty`, or `unknown` when that is empty, to + the `Dockerfile` as the build argument `VERSION`. A build argument is a value + that a `docker build` hands to the `Dockerfile`. The build stage depends on + the `lint` and `test` stages, so this also runs the tests and the linter. +- `script/cibuild` runs `script/bootstrap` and then `script/check`. Then it + builds the image as `script/docker` does. The Gitea workflow in + `.gitea/workflows/check.yml` runs it on every push. A successful run means + every check passed. The tests and the linter run twice, once in `script/check` + and again as stages of the image build. +- `script/precommit` runs from the `git` pre-commit hook, a script that `git` + runs before each commit. It runs `go mod tidy`, and fails the commit when that + changes `go.mod` or `go.sum`. Then it runs `script/check`. +- `script/install-precommit` installs `.git/hooks/pre-commit`, which runs + `script/precommit`. Every `docker build` in `script/` passes a new `CACHEBUST` build argument. In -every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps -that install system packages and download the Go modules, or right after `FROM` -in a stage with neither, and every step that lints, tests or compiles comes -after it, so every `RUN` step after it runs again each time. On an unchanged -tree Docker would otherwise serve the gate steps from its cache, and the build -would pass having run no test and no linter. The layers above that line stay in -Docker's build cache until the base image, their own lines, `go.mod` or `go.sum` -change, or until that cache is cleared; only then are the packages installed and -the modules downloaded again. Go's build cache is kept between builds on the -steps that compile, so only what changed is compiled again; the tests run with -`-count=1`, so none reports a stored result. `golangci-lint run` also keeps -golangci-lint's cache, and takes a package's findings from it only when the -package's files and those of the packages it imports, `go.mod`, the linters and -their settings in `.golangci.yml`, and the golangci-lint version are all -unchanged. +every stage of the `Dockerfile`, `ARG CACHEBUST` comes after the steps that +install system packages and download the Go modules. Every step that checks or +compiles the code comes after it. Every `RUN` step after it runs again on each +build. Without that, Docker would serve the test and lint steps from its cache +on an unchanged tree, and the build would pass without running them. The steps +above `ARG CACHEBUST` stay cached until a change to something they depend on, +such as `go.mod`. Go's build cache is kept between builds, so only what changed +is compiled again. The tests run with `-count=1`, so no test reports a stored +result. `golangci-lint run` keeps the linter's own cache as well, so an +unchanged package is not linted again. ## Build -The `script/` entrypoints above are where the implementations live; the -`Makefile` targets are shims onto them, except `build`, which carries the -compile recipe: +The `script/` entrypoints above hold the implementations. Each `Makefile` target +does nothing but call its script, except `build` and `clean`, which have +commands of their own: -- `make` / `make build` — build the `sfdupes` binary (cgo disabled); building is - the default target. -- `make bootstrap` — install the build and development dependencies. -- `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook. -- `make test` — run the test suite under the race detector, in Docker (90-second - timeout; reruns with `-v` on failure; see `script/test`); requires `docker`. -- `make lint` — run `golangci-lint` with the repo config and the gofmt check, in - Docker (see `script/lint`); requires `docker`. -- `make fmt` / `make fmt-check` — format the Go sources and the Markdown / - verify formatting without writing, on the host; requires `go` and what - `make bootstrap` installs (see `script/fmt`). -- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires - `docker` and what `make bootstrap` installs. -- `make docker` — build the Docker image, which runs the gates as build stages. -- `make hooks` — install the pre-commit hook. -- `make clean` — remove the binary. +- `make` or `make build` builds the `sfdupes` binary with cgo disabled. Building + is the default target. +- `make bootstrap` installs the build and development dependencies. +- `make setup` prepares a fresh clone: `bootstrap`, then the pre-commit hook. +- `make test` runs the test suite under the race detector, in Docker (see + `script/test`). It needs `docker`. +- `make lint` runs the linter and the `gofmt` check, in Docker (see + `script/lint`). It needs `docker`. +- `make fmt` formats the Go sources and the Markdown on the host, and + `make fmt-check` checks them without writing. Both need `go` and what + `make bootstrap` installs. +- `make check` runs `script/check`, described above, and changes nothing. It + needs `docker` and what `make bootstrap` installs. +- `make docker` builds the Docker image, which runs the tests and the linter as + build stages. +- `make hooks` installs the pre-commit hook. +- `make clean` removes the binary. + +### Constraints + +- The language is Go. The module is `sneak.berlin/go/sfdupes`, and the binary is + `sfdupes`. +- The code imports these modules beyond the standard library: + - `github.com/spf13/cobra`, for the command line; + - `github.com/schollz/progressbar/v3`, for the progress display; + - `golang.org/x/term`, to tell whether stderr is a terminal; + - `modernc.org/sqlite`, the SQLite driver; + - `golang.org/x/sys`, for the scan lock and the list of mounted filesystems. + + No other dependency outside the standard library is added. + +- The code stays pure Go, with no C code. Building on one system for another is + not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without + cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one + exception, because the race detector needs cgo (see `script/test`). +- `report` and `trees` give the same output for the same database contents, + whatever order the records were inserted in. ### Definition of done All of the following, run in this directory, must pass: -1. `make check` passes (tests, lint, `gofmt`, prettier). -2. `make docker` succeeds. -3. Smoke test — create a throwaway tree in a temp dir (never test against real - data): +1. `make check` passes. +2. `make docker` succeeds. +3. The test below behaves as described. It builds a throwaway tree in a new + directory under `/var/tmp`, which is on a persistent filesystem. On many + systems `/tmp` is a `tmpfs`, which `scan` leaves out. Never test against real + data. ```sh - d=$(mktemp -d) - export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite" + make build + d=$(mktemp -d /var/tmp/sfdupes.XXXXXX) + export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite" mkdir -p "$d/a" "$d/b" head -c 2000 /dev/urandom > "$d/a/one.bin" cp "$d/a/one.bin" "$d/b/copy.bin" @@ -1204,44 +1402,35 @@ All of the following, run in this directory, must pass: ./sfdupes report ``` - Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one - group (two dupe rows, `first` is the lexicographically smallest path); - `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/ - `t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair; - `empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; groups - ordered by size descending. + Expected from the first `report`: + - `one.bin`, `copy.bin` and `copy2.bin` form one group of two rows. `first` + is the path that sorts first. + - `t1/f1`, `t2/f1` and `t3/f1` form one group. + - `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group. + - `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`. + - `unique.bin` and `tiny3` appear nowhere. + - The groups come in order of size, largest first. - Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2 - files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal - (implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set - differs by name). + `trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`. + The row shows 2 files, 3100 bytes, and a `disk_size` that depends on the + filesystem. The two `sub` directories are left out, because the `t1` and + `t2` group implies them. `t3` appears nowhere, since one of its file names + differs. - Expected from the second `report` (after the modify/delete rescan): - `one.bin` has left its group (its content changed), so - `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is gone from the - database. + After the rescan, the second `report` shows `one.bin` out of its group, + because its content changed. `copy.bin` and `copy2.bin` remain as one pair. + `unique.bin` is gone from the database. - The test suite automates this scenario (see `scan_test.go`), plus a negative - check: `report` and `trees` operate on the database alone and never touch - the scanned filesystem. + The test suite automates this scenario (see `scan_test.go`). It also checks + that `report` and `trees` never touch the scanned filesystem. ## TODO -Tracked in [TODO.md](TODO.md). - -## Non-goals - -- No byte-for-byte compare, and no deletion or linking of duplicates. Files that - match are compared by a SHA-256 of the whole file below 50 MiB, and only by - samples at 50 MiB and over. The reports are advisory; acting on them is the - user's job. -- No persistence beyond the SQLite database described above; no export/import - formats. -- No daemon or filesystem watcher; scheduling rescans is cron's job. +Tracked in [`TODO.md`](TODO.md). ## License -MIT. See [LICENSE](LICENSE). +MIT. See [`LICENSE`](LICENSE). ## Author diff --git a/TODO.md b/TODO.md index 4dd4dff..36d6907 100644 --- a/TODO.md +++ b/TODO.md @@ -1,528 +1,763 @@ # Workflow -- take an issue from the `1.0.0` milestone on the tracker; work not yet on the - tracker gets filed as an issue first -- branch from `next` -- do the work, with tests, in small focused commits -- record it at the top of Completed Steps (`TODO.md` changes in the same commit - as the work) -- push the branch and open a PR against `next` whose title ends with - ` (closes #N)` -- an independent review gates each merge to `next`; every finding is addressed - or explicitly rebutted on the PR -- only the owner merges `next` to `main` +- Take an issue from the `1.0.0` milestone on the tracker. Work that is not on + the tracker yet is filed as an issue first. +- Branch from `next`. +- Do the work, with tests, in small focused commits. +- Record the step at the top of Completed Steps, in the same commit as the work. +- Push the branch and open a PR against `next`. Its title ends with + ` (closes #N)`. +- An independent review gates each merge to `next`. Every finding is addressed + or rebutted on the PR. +- Only the owner merges `next` into `main`. # Status -- pre-1.0 -- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues - under the `1.0.0` milestone are what remains before the tag, and this file - records history and process, not the queue +- Pre-1.0. +- The tracker is authoritative for the pre-1.0 backlog. The open issues under + the `1.0.0` milestone are what remains before the tag. This file records + history and process, not the queue. # Next Step -- take the next issue from the `1.0.0` milestone on the tracker: - https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source - of truth for what is left before 1.0.0. Individual issues are deliberately not - restated here; a copy in this file drifts out of date the moment the tracker - moves +- Take the next issue from the `1.0.0` milestone, + https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated + here, because a copy drifts as soon as the tracker moves. # Completed Steps -- re-vendor `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` from - `sneak/prompts` `next` at `0653579`: in the `lint`, `test` and `builder` - stages of the `Dockerfile` the module download, and in `builder` the - system-package install, now come before `ARG CACHEBUST`, so Docker keeps them - until the base image, the install line, `go.mod` or `go.sum` changes, and - every step that lints, tests or compiles still comes after it; - `golangci-lint run` also keeps golangci-lint's cache. README says the same - (2026-10-18, https://git.eeqj.de/sneak/sfdupes/issues/109) +Newest first. An entry ends with its date and its issue or branch, where those +are known. -- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so - `go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has - passed the build; README "Install" says so, says what `@main` and `v0.0.1` - give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the - early build without a database. When the linker set no version, - `sfdupes --version` prints the main module's version that Go recorded in the - binary, the tag or the pseudo-version of a `go install` build, and `dev` only - when neither is known (2026-10-17, - https://git.eeqj.de/sneak/sfdupes/issues/116) +- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then + what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees` + prints the report again (2026-11-06, + https://git.eeqj.de/sneak/sfdupes/issues/154) -- `scan` creates the database's directory where the database path leads: the - path with its last element cut off as text, never cleaned, so a `..` after a - symlinked directory is followed through the symlink, as when the file is - opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a - symlink to `/data/x`, it creates `/data/new` and the database in it, where it - used to create `/srv/new` and then fail (2026-10-17, +- `scan` resolves each operand to its real path, symlinks followed, before the + walk. A symlink operand is scanned at its target instead of skipped. With + `/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub` + walks `sub` on its own instead of dropping it and deleting its records + (2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144) + +- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's + blocks only seconds after the write. The count can land in under a second. + Both now say up to seconds after the write, as the comment on `diskSizeOf` in + `scan_test.go` does (2026-11-03, https://git.eeqj.de/sneak/sfdupes/issues/152) + +- The three disk size tests in `scan_test.go` no longer fail at random on ZFS. + `unix.Sync` can return before ZFS has counted a file's blocks, and one ZFS + commit can take a file through several sizes. The test that needs the size to + change waits for it, for at most 30 seconds. Each test reads the size before + and after its scan. The rewrite test rescans until the two reads match, then + compares exactly. The other two fail only on a `disk_size` outside the two + reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150) + +- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by + sentence. Every completed step is kept, with its date and its issue or branch. + The policy-compliance checklist of 2026-07-22 is folded into its step + (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135) + +- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads + it. It leads with what the program is and how to use it. A reference section + covers every setting and every subcommand. Development and testing come last. + Every statement was checked against the code on `next`. Five comments in + `main_test.go` cite "Settings", where their rules now live. One comment each + in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README + now says (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135) + +- Each record holds `disk_size`, the bytes the file takes on disk. That is its + allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already + makes. `trees` prints a `disk_size` column after `size`. It is the sum over + the `dupe` copy's files. On a compressed or sparse file `disk_size` differs + from `size`. A rescan updates the `disk_size` of an unchanged file whose size + on disk changed, without reading the file (2026-10-27, + https://git.eeqj.de/sneak/sfdupes/issues/147) + +- The database keeps the number of records and the sum of their sizes in the one + row of a `totals` table. Three triggers on `files` keep that row current. + `info` reads that row. So do `scan`'s `load` phase and the `read` phases of + `report` and `trees`, for their totals. No count reads every record any more. + The triggers make each write to `files` slower. `scan` keeps SQLite's + temporary data in memory. That wins back part of that time (2026-10-27, + https://git.eeqj.de/sneak/sfdupes/issues/146) + +- an operand at or inside a directory named `.zfs` is never scanned, whether + given alone or under another operand. `scan` warns about it, counts it as + skipped and does not walk it (2026-10-26, + https://git.eeqj.de/sneak/sfdupes/issues/142) + +- A file or directory removed while the scan runs is not an error, in the walk + or in the hash phase. The scan prints no line for it and does not count it as + skipped. When the first of several hard links to a file is gone, the hash + phase reads the next one instead. On a terminal every progress display starts + with the spinner, the hash phase's included. A phase with an exact total ends + its line with its percent and a 10-column bar at the terminal's right edge. + The bar no longer moves as the line's length changes. A line too wide for the + terminal now loses its rate before its bar. The time remaining is labelled + `remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133) + +- The `head`, `tail` and `content` columns are BLOBs, each holding the raw 32 + bytes of a SHA-256. That is half the size of the hex text they held. The + hashes stay raw from the hash phase through the database to `report` and + `trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26, + https://git.eeqj.de/sneak/sfdupes/issues/134) + +- The hash phase gives three hashes to every file that the walk records. `head` + and `tail` are the SHA-256 of its first and last 64 KiB. `content` is the + SHA-256 of the whole file under 100 MiB, and of 1 MiB at each gigabyte from + 100 MiB up. Gone are: + - the 10 MiB and 50 MiB thresholds; + - the size census, which counted the sizes of every walked file and of every + record outside the operands. A file was hashed only when the census showed + another file of its size; + - the records without hashes; + - the content phase. + + A scan has four phases, and a rescan reads only new and changed files. + `info` no longer counts the records with `head` and `tail` hashes, nor those + with a content hash, since every record has all three (2026-10-25, + https://git.eeqj.de/sneak/sfdupes/issues/132) + +- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` were copied + again from `sneak/prompts` `next` at `0653579`. Two steps now come before + `ARG CACHEBUST`: + - the module download in the `lint`, `test` and `builder` stages of the + `Dockerfile`; + - the package install in `builder`. + + Docker keeps them until one of these changes: + - the base image; + - the install line; + - `go.mod` or `go.sum`. + + Every step that checks or compiles the code still comes after + `ARG CACHEBUST`. `golangci-lint run` also keeps the linter's own cache. + README says the same (2026-10-18, + https://git.eeqj.de/sneak/sfdupes/issues/109) + +- The release rule was set. Until 1.0, each tip of `main` that passes the build + gets the next `v0.0.N` tag. `go install sneak.berlin/go/sfdupes@latest` then + installs it. `go.mod` retracts `v0.0.1`, the early build without a database. + When the linker set no version, `sfdupes --version` prints the version Go + recorded in the binary. That is the tag, or the pseudo-version of a + `go install` build. It prints `dev` only when neither is known. README + "Install" documented: + - the rule; + - what `@main` and `v0.0.1` give; + - how to check an installed binary. + + (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/116) + +- `scan` creates the database's directory where the database path leads. The + directory is the path with its last element cut off as text, so a `..` in it + stays. A `..` after a symlinked directory therefore goes through the symlink, + as it does when the file is opened. With + `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a symlink to + `/data/x`, it now creates `/data/new` and the database in it. It used to + create `/srv/new` and then fail (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/119) -- `scan` never records its own database: once the database and its `.lock` file - are open, it notes the device and inode of the database file, its `-wal` and - `-shm` files and its `.lock` file, and the walk passes over a file with one of - them wherever the operands reach it, through a symlinked directory included, - printing no line and counting nothing. A record an earlier scan left for one - of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115) +- `scan` never records its own database. Once the database and its `.lock` file + are open, it notes the device and inode of each of these: + - the database file; + - its `-wal` and `-shm` files; + - the `.lock` file. -- `scan` never records or reads anything that is not on a persistent filesystem - (`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types), - matched by the device ID of each such filesystem, listed as the scan starts - from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS, - however the walk or the content phase reaches it; no flag turns that off. - Below an operand, its walk does not enter the default excludes either, so - `sfdupes scan /` skips the kernel's view of the running system and the - temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and - `/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`, - `/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a - Windows list, not used yet. An operand at or inside a default exclude is still - walked. The scan prints a line for each path it leaves out, without counting - it as skipped, and the records there count as outside the scanned operands, as - for `-x`. `--no-default-excludes` turns off the default excludes only, and - `scan --help` lists them (2026-10-17, - https://git.eeqj.de/sneak/sfdupes/issues/124) + The walk passes over any file with the device and inode of one of them, + however it reaches the file. It prints no line for such a file and counts + nothing. A record an earlier scan left for one of them is deleted + (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115) -- the rate a progress display shows covers the items completed over the last 5 - seconds, or since the line before when stderr is not a terminal, so it falls - to 0 while nothing completes, and a burst after a wait no longer reads as a - steady rate of over 1,000 files a second; the ETA still comes from the phase's - average. Once no item has completed for 10 seconds, the word `stalled` and - where the phase waits take the rate's place, as `stalled disk/db`: `disk`, - `db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on - either, each phase giving its word where it is defined. A line too wide for - the terminal loses its bar first, then its rate or those words, so the phase, - progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts - like those of README's example, the content phase's checks and reads lose - those words at most times in minutes, and every other phase keeps them - whenever the elapsed time and the ETA are both in minutes. The content phase's - last partial batch is now written under a display of its own (2026-10-16, - https://git.eeqj.de/sneak/sfdupes/issues/125) +- `scan` never records or reads anything on a filesystem that is not persistent, + such as `tmpfs` or `proc`. Such a filesystem is one the kernel makes up or + keeps in memory. `scan` matches them by device ID, however the walk or the + content phase reaches a path. As the scan starts, it lists the mounted + filesystems of those types, from `/proc/self/mountinfo` on Linux and from the + mount list on macOS. No flag turns that off. Below an operand, the walk does + not enter the default excludes either. So `sfdupes scan /` skips the kernel's + view of the running system and the temporary directories. On Linux those are: -- new subcommand `info` prints the database's details on stdout, one per line as - a name, a tab and the value: its path and where it came from, the sizes of the - database file and of its `-wal` file when there is one, when the database file - was last changed, the schema version, the number of records, how many have - head and tail hashes and how many a content hash, and the sizes of all records - added up. It opens the database as `report` does, and shows two phases on - stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come - from one read-only transaction, the counts from one query (2026-10-16, - https://git.eeqj.de/sneak/sfdupes/issues/117) + ``` + /proc, /sys, /dev, /run, /tmp, /var/tmp + ``` + + On macOS they are: + + ``` + /dev, /private/tmp, /private/var/tmp, /private/var/vm + /System/Volumes/VM, /System/Volumes/Data + ``` + + A Windows list exists in the code but is not used. An operand at or inside a + default exclude is still walked. The scan prints a line for each path it + leaves out, without counting it as skipped. The records there count as + outside the scanned operands, as for `-x`. `--no-default-excludes` turns off + the default excludes only, and `scan --help` lists them (2026-10-17, + https://git.eeqj.de/sneak/sfdupes/issues/124) + +- A progress display's rate covers the items completed over the last 5 seconds. + When stderr is not a terminal, it covers the items since the line before. It + falls to 0 while nothing completes. A burst after a wait no longer reads as a + steady rate of over 1,000 files a second. The time remaining still comes from + the phase's average. Once no item has completed for 10 seconds, the line shows + `stalled` and where the phase waits in place of the rate, as + `stalled disk/db`. The places are: + - `disk`; + - `db`; + - `out`, for stdout; + - `cpu`. + + A phase that may wait in two places names both. Each phase's word is set in + the code that defines the phase. A line too wide for the terminal loses its + bar first, then its rate or those words. Four parts stay whole: + - the phase; + - the progress; + - the elapsed time; + - the ETA, the time remaining. + + The content phase had one display for its checks and one for its reads. Both + lost those words on an 80-column terminal, at counts like those of README's + example. That happened for most elapsed times and ETAs in minutes. Every + other phase kept them whenever the elapsed time and the ETA were both in + minutes. The content phase writes its last partial batch, the records not + yet written, under a display of its own (2026-10-16, + https://git.eeqj.de/sneak/sfdupes/issues/125) + +- New subcommand `info` prints the database's details on stdout, one per line as + a name, a tab and the value: + - the path and where it came from; + - the sizes of the database file and of its `-wal` file, when there is one; + - when the database file was last changed; + - the schema version; + - the number of records; + - how many records have `head` and `tail` hashes; + - how many records have a content hash; + - the sizes of all records added up. + + It opens the database as `report` does and shows two phases on stderr, + `[1/2] open` and `[2/2] count`. The schema version and the counts come from + one read-only transaction, a read of the database at one moment. The counts + come from one query (2026-10-16, + https://git.eeqj.de/sneak/sfdupes/issues/117) - `report` shows its work on stderr in two phases, `[1/2] open` and - `[2/2] read`, with the progress display `scan` and `trees` use, from the - moment it starts until the summary line: `open` says it is opening the - database, also while it waits on a busy one; `read` counts the records, then - reads them against that count in signature order, which SQLite takes from the - `files_signature` index without sorting, so the first record comes at once. - `report` forms the duplicate groups itself, holding only those of the size it - is reading, and writes them ordered by first path before it reads the next - size. stdout and the summary line are unchanged, and SQLite no longer needs a - temporary file to sort in (2026-10-16, + `[2/2] read`, with the progress display `scan` and `trees` use. The display + runs from the moment `report` starts until the summary line. `open` says it is + opening the database, also while it waits on a busy one. `read` counts the + records, then reads them against that count in signature order. SQLite takes + that order from the `files_signature` index without sorting, so the first + record comes at once. `report` forms the duplicate groups itself and holds + only those of the size it is reading. It writes them ordered by first path + before it reads the next size. stdout and the summary line are unchanged, and + SQLite no longer needs a temporary file to sort in (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/114) -- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`, - `[3/4] group` and `[4/4] write`, each with the progress display `scan` uses, - from the moment it starts until the summary line: `open` says it is opening - the database, also while it waits on a busy one; `read` counts the records, - then reads them against that count; `group` counts the directories it groups, - and `write` the groups it writes. A display's rate names what its phase - counts: files, records, directories or groups. stdout and the summary line are - unchanged. When stdout and stderr are the same file, as one terminal is, none - of stdout's lines shares a line with the display; when they are not, the - display keeps being drawn while stdout waits on a slow reader (2026-10-16, - https://git.eeqj.de/sneak/sfdupes/issues/113) +- `trees` shows its work on stderr in four phases, from `[1/4] open` to + `[4/4] write`, each with the progress display `scan` uses. The display runs + from the moment `trees` starts until the summary line. `open` says it is + opening the database, also while it waits on a busy one. `read` counts the + records, then reads them against that count. `group` counts the directories it + groups, and `write` the groups it writes. A display's rate names what its + phase counts: + - files; + - records; + - directories; + - groups. -- each `scan` phase's progress display names the phase and its place among the - five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed - time and ETA: `load` counts the records first, the walk's total is estimated - from the records under the operands, and the content phase counts the records - it checks from SQLite's grouping alone, then says it is searching the database - until the first one is returned. On a terminal a phase with an exact total - draws its line followed by the library's bar, and any other a spinner followed - by the line; every display is drawn again several times a second. No line - reaches the terminal's last column, however narrow, also after the terminal is - made narrower during a phase: a line too wide leaves out its rate, then its - bar, keeping the phase, progress, elapsed time and ETA whole, and is cut short - only when those alone do not fit, as below about 70 columns late in a large - scan. In plain mode the line is printed every 5 seconds whether or not an item - completed (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/112) + stdout and the summary line are unchanged. When stdout and stderr are the + same file, as one terminal is, no line of stdout shares a line with the + display. When they are not, the display keeps being drawn while stdout waits + on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113) + +- Each `scan` phase's progress display names the phase and its place among the + five, as `[2/5] walk`. It runs from the moment that the phase starts. It + shows: + - the progress; + - the elapsed time; + - the ETA, the time remaining. + + `load` counts the records first. The walk's total is estimated from the + records under the operands. The content phase takes its total from SQLite's + grouping alone. Until the first record is returned, it says it is searching + the database. On a terminal a phase with an exact total draws its line + followed by the bar of the progress-bar library. Any other phase draws a + spinner followed by the line. Every display is drawn again several times a + second. No line reaches the terminal's last column, however narrow the + terminal is, even after it is made narrower during a phase. A line too wide + leaves out its rate, then its bar. The rest stays whole. The line is cut + short only when the rest alone does not fit, as below about 70 columns late + in a large scan. When stderr is not a terminal, the line is printed every 5 + seconds whether or not an item completed (2026-10-16, + https://git.eeqj.de/sneak/sfdupes/issues/112) - `scan`, `report` and `trees` first print a line on stderr naming the database - they use, as an absolute path, and whether that path is the default or comes - from `SFDUPES_DATABASE`; `scan` says when the database is new. README + they use, as an absolute path. The line says whether that path is the default + or comes from `SFDUPES_DATABASE`. `scan` says when the database is new. README "Database" says when `scan` writes each file (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/111) -- re-vendor the canonical files and model scripts from `sneak/prompts` `next` at - `e3f79c4`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile` - that write no image, and `make test` runs the suite under the race detector, - so `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race` are - gone; every `docker build` in `script/` passes a new `CACHEBUST` value, which - every stage of the `Dockerfile` declares first, so every `RUN` step runs on - every build, while the steps that compile keep Go's build cache and the tests - run with `-count=1`; prettier runs on the host, from the node and yarn - `script/bootstrap` installs, so the `prettier` and `markdown` stages are gone - and the build stage installs `git` and `make` itself; a new push cancels the - workflow's older run on the same branch, and a run stops after 20 minutes; - `.claude/settings.json` is deleted (2026-10-15, - https://git.eeqj.de/sneak/sfdupes/issues/95). Since - https://git.eeqj.de/sneak/sfdupes/issues/109 the module download in `lint`, - `test` and the build stage, and the install of `git` and `make` in the build - stage, come before `ARG CACHEBUST`, and Docker keeps them between builds; - every step that lints, tests or compiles still runs on every build +- The canonical files and model scripts were copied again from `sneak/prompts` + `next` at `e3f79c4`: + - `golangci-lint` v2.14.0; + - lint and test are phases of the `Dockerfile` that write no image, and + `make test` runs the suite under the race detector. Gone are + `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race`; + - every `docker build` in `script/` passes a new `CACHEBUST` value. Every + stage of the `Dockerfile` declares it first, so every `RUN` step runs on + every build. The steps that compile keep Go's build cache, and the tests + run with `-count=1`; + - `prettier` runs on the host, from the `node` and `yarn` that + `script/bootstrap` installs. The `prettier` and `markdown` stages are + gone, and the build stage installs `git` and `make` itself; + - a new push cancels the workflow's older run on the same branch, and a run + stops after 20 minutes; + - a coding agent's settings file is deleted. -- with `-x`, an operand under another operand is walked on its own instead of - being dropped when a directory from it (from its directory, for a regular - file) up to but not including that operand is on a different filesystem than - that operand, whose walk then cannot reach it, and the records of files gone - from below it are deleted (2026-10-14, + Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before + `ARG CACHEBUST`: + - the module download in `lint`, `test` and the build stage; + - the install of `git` and `make` in the build stage. + + Docker keeps them between builds. Every step that checks or compiles the + code still runs on every build (2026-10-15, + https://git.eeqj.de/sneak/sfdupes/issues/95) + +- With `-x`, an operand under another operand is no longer dropped when a + directory between the two, the inner one included, is on a different + filesystem than the outer one. The inner operand is then walked on its own. + For a regular-file operand the check starts at its directory. The records of + files gone from below it are deleted (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/106) -- with `-x`, `scan` keeps the records below a directory on another filesystem - under its operand, and counts their sizes in the size census, instead of - deleting them (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99) +- With `-x`, `scan` keeps the records below a directory on another filesystem + under its operand, instead of deleting them. It counts their sizes in the size + census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99) -- the hash and content phases open each file without blocking and check that it - is still a regular file, so one replaced by a FIFO or device node after it was - statted is warned about and skipped instead of hanging the scan (2026-10-14, +- The hash and content phases open each file without blocking and check that it + is still a regular file. A file that is now a FIFO or device node is warned + about and skipped instead of hanging the scan (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/100) -- the scan lock file is named from the database's path with symlinks resolved, - after `scan` creates the database file if it is missing, so a scan through a - symlink to the database fails while another scan holds the lock (2026-10-14, - https://git.eeqj.de/sneak/sfdupes/issues/101) +- The scan lock file is named from the database's path with symlinks resolved, + after `scan` creates the database file if it is missing. A scan through a + symlink to the database therefore fails while another scan holds the lock + (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/101) -- on a terminal, a warning for the last file of a phase no longer erases the - phase's finished bar: the bar is drawn again below the warning (2026-10-14, +- On a terminal, a warning for the last file of a phase no longer erases the + phase's finished bar. The bar is drawn again below the warning (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/102) -- `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus - `mtime_nsec`, and compares it at that resolution, so a same-size rewrite - within the same second is re-hashed (2026-10-07, +- `scan` records the mtime to the nanosecond, as whole seconds in `mtime` plus + `mtime_nsec`, and compares it at that resolution. A same-size rewrite within + the same second is hashed again (2026-10-07, https://git.eeqj.de/sneak/sfdupes/issues/12) -- cut the narration from `TODO.md` Completed Steps and from the comments in +- The narration was cut from `TODO.md` Completed Steps and from the comments in `script/`, `Dockerfile` and `Dockerfile.lint` (gone since - https://git.eeqj.de/sneak/sfdupes/issues/95); §Workflow now branches from and - merges to `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49) + https://git.eeqj.de/sneak/sfdupes/issues/95). The Workflow section now + branches from and merges to `next` (2026-10-04, + https://git.eeqj.de/sneak/sfdupes/issues/49) - `make test-race` ran the test suite under the race detector in a cgo-enabled - container, outside `make check` (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/18). Since + container, outside `make check`. Since https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite - under the race detector, in the `Dockerfile`'s Debian-based `test` phase, and - `make test-race` is gone + under the race detector, in the `Dockerfile`'s Debian-based `test` phase. + `make test-race` is gone (2026-10-04, + https://git.eeqj.de/sneak/sfdupes/issues/18) -- a bare `docker build .` failed, naming `script/cibuild` and `script/docker`, - rather than serve the gates from cache (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/39). Since +- A bare `docker build .` failed, naming `script/cibuild` and `script/docker`, + rather than serve the gates, the lint and test steps, from cache. Since https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as - `REPO_POLICIES.md` requires, and may serve the gates from cache; the builds in - `script/` pass a new `CACHEBUST` value, so theirs always run + `REPO_POLICIES.md` requires, and may serve the gates from cache. The builds in + `script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04, + https://git.eeqj.de/sneak/sfdupes/issues/39) -- `make fmt` and `make fmt-check` run prettier over all Markdown, and CI checks - it; all Markdown reformatted (2026-10-04, +- `make fmt` and `make fmt-check` run `prettier` over all Markdown, and the + Gitea workflow checks it. All Markdown was reformatted (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/19) - `script/lint` writes no image, so a run no longer leaves an untagged one behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48) -- tests cover a missing database, `scan` keeping stdout empty, its skip warning, - the `report` and `trees` summary lines, and every subcommand going through - `runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16) +- Tests cover: + - a missing database; + - `scan` keeping stdout empty; + - its skip warning; + - the `report` and `trees` summary lines; + - every subcommand going through `runE`. -- `.golangci.yml` replaced with the current canonical copy, which uses + (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16) + +- `.golangci.yml` was replaced with the current canonical copy, which uses `gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/26) -- a test fails when either `hashWorker` cancellation check in `scan.go` is +- A test fails when either `hashWorker` cancellation check in `scan.go` is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83) -- a database path holding `?`, `#` or `%` opens exactly the file it names +- A database path holding `?`, `#` or `%` opens exactly the file it names (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55) - `scan` rejects `--workers` below 1 as a usage error instead of running single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10) -- a test fails when either walk cancellation check in `scan.go` is removed +- A test fails when either walk cancellation check in `scan.go` is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81) -- test that `scan` refuses a database with another schema version (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/64) +- A test checks that `scan` refuses a database with another schema version + (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64) -- correct four inaccurate comments in `cancel_test.go` and rename - `walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04, +- Four inaccurate comments in `cancel_test.go` were corrected, and + `walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/33) -- test the `-x` filesystem-boundary rules in `subdirJob` (2026-10-04, +- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/17) -- `scan` creates the schema in one transaction; a version-0 database with a +- `scan` creates the schema in one transaction. A version-0 database with a `files` table is refused with a clear schema-version error (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/11) -- README documents install, Docker, a daily cron scan and how to read and check - the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54) +- README documents install, Docker, a daily `cron` scan and how to read and + check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54) -- the `Dockerfile` build stage kept the Go module cache out of `builder`'s home - and copied the sources with `--chown`, so no `chown -R` walked them - (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43). Since +- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home + and copied the sources with `--chown`, so no `chown -R` walked them. Since https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and - nothing changes owner: the build stage only compiles, as root, and the tests - run as `nobody` in the `test` phase + nothing changes owner. The build stage only compiles, as root, and the tests + run as `nobody` in the `test` phase (2026-10-04, + https://git.eeqj.de/sneak/sfdupes/issues/43) -- `--version` prints `sfdupes VERSION` to stdout; README documents it and +- `--version` prints `sfdupes VERSION` to stdout. README documents it and `--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15) -- `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has hashed, - deletes nothing more, exits 1 (2026-10-04, +- `scan` stops cleanly on `SIGINT` or `SIGTERM`: it commits what it has hashed, + deletes nothing more, and exits 1 (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/5) -- `report` and `trees` stream the records instead of holding them all in memory; - the schema gains the `files_signature` index (2026-10-04, +- `report` and `trees` stream the records instead of holding them all in memory. + The schema gains the `files_signature` index (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/14) -- progress prints at once on a non-terminal, uses a real terminal test, and - prints warnings through a spinner instead of racing its redraw (2026-10-03, +- Progress prints at once on a non-terminal and uses a real terminal test. + Warnings print through a spinner instead of racing its redraw (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/13) -- warn about and skip symlink, socket, FIFO, device and `.zfs` operands, keeping - the records beneath them (2026-10-03, - https://git.eeqj.de/sneak/sfdupes/issues/9) +- These operands are warned about and skipped, and the records beneath them are + kept: + - symlinks; + - sockets; + - FIFOs; + - device nodes; + - `.zfs` directories. + + (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/9) - `scan` holds a lock on a lock file beside the database for its whole run, so a second `scan` fails at once with exit 1 (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/53) -- test stdout write failures in `report` and `trees`; README states that - `| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null` +- Tests cover stdout write failures in `report` and `trees`. README states that + `| head` ends `sfdupes` by `SIGPIPE` and that `>&-` writes to `/dev/null` (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30) - `report` and `trees` open the database read-only, and `scan` leaves it out of - WAL mode, so reading needs only read access (2026-10-03, closes + WAL mode, so reading needs only read access (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/8) -- escape tabs, newlines, carriage returns and backslashes in report, trees and - warning paths; the root directory's path is `/` (2026-10-03, - https://git.eeqj.de/sneak/sfdupes/issues/7) +- Four characters are escaped in `report`, `trees` and warning paths: + - tab; + - newline; + - carriage return; + - backslash. -- stamp the git tag or short commit in a plain `docker build .` instead of `dev` - (2026-10-02, branch `next`, closes - https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git` - without `.git/config`; the build stage stamps the `VERSION` build argument, - else `git describe --tags --always`, and fails if the context carries `.git` - and the version is still empty, `dev` or `unknown`. CI checks out the full - history (`fetch-depth: 0`) so it stamps the same value as `make build`. + The root directory's path is `/` (2026-10-03, + https://git.eeqj.de/sneak/sfdupes/issues/7) -- replace the 1 KiB end-window sampling with the head/tail plus content-hash - ladder (2026-09-22, branch `next`, closes - https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection" - documents every rung. A file under 10 MiB is hashed in full, and its `head`, - `tail` and `content` all hold that hash. A larger file gets only its 64 KiB - `head` and `tail` in the hash phase; the content phase, after the update - phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1 - MiB samples at or above) only when its size, `head` and `tail` match another - record's from this scan or an earlier one, and never reads a file gone or - changed since its record was written. `report` and `trees` leave out any - record without a `content` hash. The `content` column is part of the version 1 - schema. +- A plain `docker build .` stamps the `git` tag or short commit instead of + `dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage + stamps the `VERSION` build argument, else `git describe --tags --always`. It + fails if the context carries `.git` and the version is still empty, `dev` or + `unknown`. The Gitea workflow checks out the full history (`fetch-depth: 0`), + so it stamps the same value as `make build` (2026-10-02, branch `next`, + https://git.eeqj.de/sneak/sfdupes/issues/67) -- remove the dead `files.dat` references from `Makefile`, `.gitignore` and - `.dockerignore` (2026-09-21, branch `next`, closes +- The 1 KiB end-window sampling was replaced with the hashes chosen by file + size, and README "Duplicate detection" documented every size threshold. A file + under 10 MiB was hashed in full, and its `head`, `tail` and `content` all held + that hash. A larger file got only its 64 KiB `head` and `tail` in the hash + phase. The content phase ran after the update phase. It read a file for + `content` only when its size, `head` and `tail` matched another record's, from + this scan or an earlier one. `content` was the whole file below 50 MiB, and + gigabyte-spaced 1 MiB samples at or above. It never read a file gone or + changed since its record was written. `report` and `trees` left out any record + without a `content` hash. The `content` column is part of the version 1 + schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132 + (2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61) + +- The dead `files.dat` references were removed from `Makefile`, `.gitignore` and + `.dockerignore` (2026-09-21, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/22) -- fix the lint-image pin comments and `FROM` form in `Dockerfile` and - `Dockerfile.lint` (2026-08-10, branch `next`, closes - https://git.eeqj.de/sneak/sfdupes/issues/25): both pins became the policy +- The lint-image pin comments and `FROM` form in `Dockerfile` and + `Dockerfile.lint` were fixed. Both pins became the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`, without the false `(Debian-based)` note or the tag. Since https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase - holds the only golangci-lint pin, in that form, so `Dockerfile.lint` and - `script/verify-lint-image-pin`, which compared the two pins, are gone. + holds the only `golangci-lint` pin, in that form. `Dockerfile.lint` is gone, + and so is `script/verify-lint-image-pin`, which compared the two pins + (2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/25) -- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10, - branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the +- All linting ran in Docker, via `Dockerfile.lint` and `script/lint`. Per the owner ruling the linter is never installed on a host, and `golangci-lint config verify` runs before `golangci-lint run`. `script/bootstrap` stopped installing or pinning the linter, and `script/verify-linter-pin` was retired. Since https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the - `Dockerfile`'s `lint` phase, which `script/lint` builds alone and the build - stage depends on through `COPY --from=lint /src/go.sum /dev/null`; + `Dockerfile`'s `lint` phase. `script/lint` builds that phase alone, and the + build stage depends on it through `COPY --from=lint /src/go.sum /dev/null`. `Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside - an image build may run docker, so the phase calls `golangci-lint` directly. + an image build may run `docker`, so the phase calls `golangci-lint` directly + (2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/46) -- install the Docker build stage's prerequisites by running `script/bootstrap` - instead of `apk add --no-cache make` inline (2026-08-09, branch - `dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42), - with `script/verify-linter-pin` failing the build unless the linter copied - from the lint stage was the version `script/bootstrap` pinned. Builds then - took up to 5m14s cold, mostly in a `chown -R` of the module cache, filed as - https://git.eeqj.de/sneak/sfdupes/issues/43. Since +- The Docker build stage's prerequisites were installed by running + `script/bootstrap` instead of `apk add --no-cache make` inline. + `script/verify-linter-pin` failed the build unless the linter copied from the + lint stage was the version that `script/bootstrap` pinned. Builds then took up + to 5m14s from an empty cache, mostly in a `chown -R` of the module cache, + filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and - `make` with `apk add --no-cache` and only compiles; the `lint` and `test` - phases are the gates -- bust the Docker layer cache for the gate steps, so `script/cibuild` and - `script/docker` cannot report a green they did not earn (2026-08-09, branch - `cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the - `Dockerfile` copies the tree before its gates, so on an unchanged tree Docker - served them from cache and the build exited 0 having run nothing. The fix was - a `CHECK_EPOCH` build argument; since - https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/` - passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile` - declares before every step that lints, tests or compiles. Run as root, the - tests fail `TestScanHardlinkRunFailsTogether`, because root reads through the - `chmod(0)` the test relies on, so the `test` phase runs them as `nobody` -- check the installed golangci-lint version in `script/bootstrap` instead of - only its presence (2026-08-09, branch `bootstrap-version-check`, closes - https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in - `GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and - any installed version that is not the pin — older, newer, absent or - unparseable — is reinstalled. `go install` writes into `GOBIN` (or - `GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so - bootstrap re-reads the effective version after installing and, on a mismatch, - prints both paths and both versions and exits non-zero; it does not reorder - `PATH` or delete anyone's binary. The `--version` call keeps its stderr and is - bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep - presence-only checks. Verified by bootstrapping this host from v2.10.1 to - v2.12.2 and again to a no-op, and by stub runs of the script under `dash` - covering a thirteen-input version-parse matrix, a shadowed install that must - exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged - binary that must hit the timeout; `make check` and `make lint` are clean at - v2.12.2, so v2.10.1 was not hiding any findings on `main` -- unwind the hash worker pool on the error path (2026-08-09, branch - `hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the - pool is now an owned, context-aware `hashPool`: every blocking send in the - feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path - out, and `hashPhase` defers `pool.stop()`, which cancels and then drains - `results` until the last goroutine has exited — draining is what frees a - worker already parked on a send. `ctx` is threaded from `cmd.Context()` - through `runScan`, `syncScan`, both worker pools and the whole database layer, - as the first parameter everywhere. The walk pool gets the same treatment plus - a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size - census, and every file it never reached looks vanished to the update phase. - That phase's own `BeginTx` also fails on the cancelled context before deleting - anything, but the guard is the barrier that still holds once an interrupted - scan may commit what it has. Tests drive `run(scan)` against a database whose - insert trigger aborts and assert that the scan fails instead of hanging and - that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others - cancel a scan part-way through the walk, deterministically, by counting its - own consultations of `ctx.Done()`, and assert that it stops at the guard - holding a partial census and a still-populated record index, with every record - intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`, - `feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation - branches of both pools -- guarantee the database is closed on every fatal exit path (2026-08-09, branch - `db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4): - `fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and - with it the SQLite WAL checkpoint — now actually runs when a subcommand fails; - `runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return - errors instead. The single exit point is `run` in `main.go`: it maps a - `fatalError` (anything a subcommand returned) to exit 1 and cobra's own - argument and flag errors to exit 2, which keeps a runtime failure from being - reported as a usage error or printing the usage text. New `main_test.go` - drives the CLI in-process and asserts the exit codes from README §Error - handling plus the stdout/stderr split, including that a fatal error raised - after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`, - `report` or `trees` -- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch - `golangci-v2.12.2`, merged as `38a01bd`, closes - https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the - `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and - replaced `.golangci.yml` with the canonical file — the linter settings (`lll`, - `funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per - the v2 schema, so they are actually applied; no new lint findings surfaced -- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints - like the other managed repos (2026-07-26, commit `3abeacf`, closes - https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints - exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`, - `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`) - and every Makefile target is now a thin shim over them -- make the binary the default Make target (2026-07-24, branch - `make-default-target`): plain `make` now builds `sfdupes` (previously it ran - `check` plus `build`); `make build` remains as an alias -- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch - `scan-wide-phases`): all operands seed the shared walk pool and every pass - runs once over the whole scan, so totals and ETAs are scan-global; the - per-operand walk/hash/update cycles and their stderr announcements are gone; - the update pass commits in batched transactions — the filesystem is - authoritative and the database an eventually-consistent reflection, so - scan-level atomicity is not required -- split the stat pass back out of the walk (2026-07-24, branch - `parallel-phases`): phases are strictly sequential again — walk, stat, hash, - update per operand — with parallelism only inside each phase; the walk - enumerates paths with per-directory workers and the stat pass lstats them with - per-file workers, restoring the exact total/ETA stat bar -- announce each operand on stderr before its passes (2026-07-24, branch - `scan-operand-progress`): with per-operand walk/hash/update cycles, a - multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the - whole run's -- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single - goroutine and took hours at ~20M files on a busy pool (observed: 22M files in - 4h on a ZFS server); it is now a per-directory worker-pool traversal that - records size/mtime during the walk (folding away the separate stat pass, - halving metadata I/O), and each `PATH` operand commits in its own transaction - so an interrupted scan keeps completed operands + `make` with `apk add --no-cache` and only compiles. The `lint` and `test` + phases are the gates (2026-08-09, branch `dockerfile-bootstrap`, + https://git.eeqj.de/sneak/sfdupes/issues/42) -- persistent scan database (2026-07-24, branch `persistent-database`): `scan` - now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays - disabled) keyed by absolute path that survives between runs — a rescan hashes - only new or changed files (by mtime/size), deletes records for files vanished - from under the scanned operands, and leaves records outside them untouched, so - `scan` can be cronned daily; `report` and `trees` read the database (no - positional arguments) instead of a scan stream. Database at - `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL +- The Docker layer cache is defeated for the gate steps, so `script/cibuild` and + `script/docker` cannot report a pass they did not earn. The `Dockerfile` + copies the tree before its gates. On an unchanged tree Docker served them from + cache, and the build exited 0 without running them. The fix was a + `CHECK_EPOCH` build argument. Since + https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/` + passes a new `CACHEBUST` value instead. Every stage of the `Dockerfile` + declares it before every step that checks or compiles the code. Run as root, + the tests fail `TestScanHardlinkRunFailsTogether`, because root reads through + the `chmod(0)` the test relies on. So the `test` phase runs them as `nobody` + (2026-08-09, branch `cibuild-cache-bust`, + https://git.eeqj.de/sneak/sfdupes/issues/32) + +- `script/bootstrap` checked the installed `golangci-lint` version instead of + only its presence. The version lived only in `GOLANGCI_LINT_VERSION`, with the + `go install` module reference derived from it. Any installed version that was + not the pin was reinstalled, whether: + - older; + - newer; + - absent; + - unparseable. + + `go install` writes into `GOBIN`, or `GOPATH/bin`, while `make lint` ran the + first `golangci-lint` on `PATH`. So bootstrap re-read the effective version + after installing. On a mismatch it printed both paths and both versions and + exited non-zero. It did not reorder `PATH` or delete anyone's binary. The + `--version` call kept its stderr and was bounded by `timeout(1)` where that + exists. `git`, `make` and `go` kept presence-only checks. The version check + was verified by bootstrapping this host from v2.10.1 to v2.12.2, and again + to a no-op. Stub runs of the script under `dash` covered: + - a thirteen-input version-parse matrix; + - a shadowed install, which must exit non-zero; + - an install destination not on `PATH`; + - `GOBIN` set; + - a wedged binary, which must hit the timeout. + + `make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been + hiding findings on `main`. Superseded by + https://git.eeqj.de/sneak/sfdupes/issues/46, since the linter is no longer + installed on a host (2026-08-09, branch `bootstrap-version-check`, + https://git.eeqj.de/sneak/sfdupes/issues/24) + +- The hash worker pool unwinds on the error path. The pool is an owned, + context-aware `hashPool`. Every blocking send in the feeder and the workers + selects on `ctx.Done()`, and `jobs` is closed on every path out. `hashPhase` + defers `pool.stop()`, which cancels and then drains `results` until the last + goroutine has exited. Draining is what frees a worker parked on a send. `ctx` + is passed from `cmd.Context()` as the first parameter, through: + - `runScan`; + - `syncScan`; + - both worker pools; + - the whole database layer. + + The walk pool gets the same treatment, plus a `ctx.Err()` guard after the + walk. A cancelled walk yields a partial size census, and every file that it + never reached looks vanished to the update phase. That phase's own `BeginTx` + also fails on the cancelled context before deleting anything. The guard is + still the barrier once an interrupted scan may commit what it has. Tests + drive `run(scan)` against a database whose insert trigger aborts. They + assert that the scan fails instead of hanging, and that the goroutine count + returns to its pre-scan baseline. Other tests cancel a scan part-way through + the walk, deterministically, by counting the scan's own checks of + `ctx.Done()`. They assert that it stops at the guard with every record + intact. It then holds a partial size census and a still-populated record + index. Direct tests cover the remaining cancellation branches of both pools. + They test: + - `sendEvent`; + - the walk workers; + - `dispatchDirs`; + - `feedHashJobs`; + - `hashWorker`; + - `hashPhase`. + + (2026-08-09, branch `hash-pool-cleanup`, + https://git.eeqj.de/sneak/sfdupes/issues/6) + +- The database is closed on every fatal exit path. `fatalf` and its `os.Exit(1)` + are gone. So the deferred `db.Close()` runs when a subcommand fails, and with + it the SQLite WAL checkpoint. These return errors instead: + - `runScan`; + - `runReport`; + - `runTrees`; + - `loadRecords`; + - `resolveRoots`. + + The single exit point is `run` in `main.go`. It maps a `fatalError`, + anything a subcommand returned, to exit 1, and cobra's own argument and flag + errors to exit 2. So a runtime failure is never reported as a usage error, + nor printed with the usage text. `main_test.go` drives the CLI in-process + and asserts the exit codes from README "Error handling" and the + stdout/stderr split. It also checks that a fatal error after the database is + open leaves no `-wal` or `-shm` file behind, for `scan`, `report` or `trees` + (2026-08-09, branch `db-close-on-fatal`, + https://git.eeqj.de/sneak/sfdupes/issues/4) + +- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as + `38a01bd`. The pinned linter in the `Dockerfile` lint stage and + `script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was + replaced with the canonical file. Four thresholds now live under + `linters.settings`, as the v2 schema requires, so they are applied: + - `lll`; + - `funlen`; + - `cyclop`; + - `dupl`. + + No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`, + https://git.eeqj.de/sneak/sfdupes/issues/3) + +- `Makefile` targets were converted to scripts-to-rule-them-all `script/` + entrypoints, like the other managed repos. All 12 entrypoints exist: + - `bootstrap`, `setup` and `projectname`; + - `test`, `lint`, `fmt`, `fmt-check` and `check`; + - `docker`, `cibuild`, `precommit` and `install-precommit`. + + Every `Makefile` target does nothing but call its script (2026-07-26, commit + `3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1) + +- The binary is the default `make` target. Plain `make` builds `sfdupes`, where + it used to run `check` plus `build`. `make build` remains as an alias + (2026-07-24, branch `make-default-target`) + +- The phases became scan-wide, with concurrent operands and batched updates. All + operands seed the shared walk pool, and every pass runs once over the whole + scan. So the totals and ETAs cover the whole scan. The per-operand walk, hash + and update cycles and their stderr announcements are gone. The update pass + writes in batches, each in its own transaction. The filesystem is + authoritative and the database follows it, so one scan's changes need not land + all at once (2026-07-24, branch `scan-wide-phases`) + +- The stat pass was split back out of the walk. Phases are strictly sequential + again, per operand, with parallelism only inside each phase: + - walk; + - stat; + - hash; + - update. + + The walk enumerates paths with per-directory workers, and the stat pass + lstats them with per-file workers. That restores the exact total and ETA of + the stat bar (2026-07-24, branch `parallel-phases`) + +- Each operand is announced on stderr before its passes. With per-operand walk, + hash and update cycles, a multi-operand run such as `scan /srv/*` showed pass + totals that looked like the whole run's (2026-07-24, branch + `scan-operand-progress`) + +- The walk became parallel. The walk pass was a single goroutine and took hours + at about 20 million files on a busy pool. One ZFS server took 4 hours for 22 + million files. It is now a walk by a pool of per-directory workers that + records size and mtime as it goes. That folds away the separate stat pass and + halves the metadata reads. Each `PATH` operand commits in its own transaction, + so an interrupted scan keeps completed operands (2026-07-24, branch + `parallel-walk`) + +- `scan` gained a persistent database. It maintains an SQLite database that + survives between runs, with the absolute path as each record's key. The driver + is `modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only + new or changed files, by mtime and size. It deletes the records of files + vanished from under the scanned operands and leaves records outside them + untouched. So `scan` can run daily from `cron`. `report` and `trees` read the + database, with no positional arguments, instead of a scan stream. The database + is at `/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL journaling plus a single-transaction update keep a report run during a scan - safe -- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`, - and push `main` plus tags (2026-07-23) -- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required - `PATH...` operands via cobra flags replacing the `/srv` `-root` default; new - `-x`/`--one-file-system` flag (GNU convention) to stop at filesystem - boundaries, which are crossed by default -- bring the repo into full policy compliance (2026-07-23, branch - `repo-policy-compliance`; checklist below) -- `git init` with README-only first commit; code baseline committed on `main` - (2026-07-22) -- implement `scan`, `report`, and `trees` subcommands (pre-git history) + safe (2026-07-24, branch `persistent-database`) + +- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, was added. `v0.0.1` + was tagged, and `main` and the tags were pushed (2026-07-23) + +- The `scan` command line was reworked. Required `PATH...` operands, parsed by + cobra, replace the `/srv` `-root` default. The new `-x`, `--one-file-system` + flag follows the GNU convention and stops at filesystem boundaries, which the + walk crosses by default (2026-07-23, branch `scan-required-paths`) + +- The repo was brought into compliance with `REPO_POLICIES.md` (the version of + 2026-07-06), the existing repo checklist and the Go styleguide. The audit of + 2026-07-22 found these gaps, and the branch closed them: + - `.gitignore`, with the standard patterns for secrets and for OS, editor + and Go files. The compiled `sfdupes` binary and `files.dat` had been + sitting untracked in the tree; + - `.editorconfig`; + - `LICENSE` and the README License section. MIT was assumed from house + convention, for the owner to confirm; + - `REPO_POLICIES.md` in the repo root; + - the canonical `.golangci.yml`, with its 150 findings fixed, so that + `make lint` is clean; + - the `Makefile` targets the policy requires: + - `test` and `lint`; + - `fmt` and `fmt-check`; + - `docker` and `hooks`. + + `check` depended on `build`, which wrote the binary. `make check` must + not modify files, so `check` no longer depends on `build`; + + - a test suite, which the policy required with a 30-second timeout and the + conditional `-v` rerun. At 64% coverage, it covers: + - parsing and grouping; + - digests and suppression; + - hashing and the scan pipeline; + - a multistage `Dockerfile` and `.dockerignore`. The `Dockerfile` had: + - hash-pinned images; + - a lint stage that failed fast; + - a build stage that ran `make check`; + - `.gitea/workflows/check.yml`, running `docker build .` on push with the + checkout action pinned by commit SHA; + - the README sections the policy requires. The first is a Description whose + first line names the project, its purpose and category, its license and + its author. Then come Getting Started and Rationale, and at the end TODO, + License and Author; + - the stale README non-goal "no git repository setup and no CI" removed; + - the pre-commit hook, installed with `make hooks` once that target existed. + + The code was already `gofmt`-clean, so no standalone formatting commit was + needed (2026-07-23, branch `repo-policy-compliance`) + +- `git init` was run, with a README-only first commit. The code baseline was + committed on `main` (2026-07-22) + +- The `scan`, `report` and `trees` subcommands were implemented (pre-git + history) # Future Steps -- possible later features (explicitly out of scope per README): full-content - verification of candidates, removal-script helpers +- Possible later features, out of scope per README, are a byte-for-byte check of + candidates and helper scripts for removal. -# Repo Policy Compliance +# Accepted Divergences -Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo -checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone -formatting commit is needed. - -- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit - untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns -- [x] `.editorconfig` missing -- [x] `LICENSE` missing and README has no License section (MIT assumed from - house convention — user to confirm) -- [x] `REPO_POLICIES.md` missing from repo root -- [x] `.golangci.yml` missing (install canonical copy); code must then pass - `make lint` (150 findings fixed; `make lint` is clean) -- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`, - `docker`, `hooks`; `check` currently depends on `build`, which writes the - binary (`make check` must not modify files) -- [x] no tests — `go test ./...` has nothing to run; policy requires real tests - with a 30-second timeout and the conditional `-v` rerun pattern (suite - covers parsing, grouping, digests, suppression, hashing, and the scan - pipeline; 64% coverage) -- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast - lint stage, build stage running `make check` -- [x] `.dockerignore` missing -- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout - action pinned by commit SHA) -- [x] README lacks required sections: Description first line - (name/purpose/category/license/author), Getting Started, Rationale, TODO, - License, Author -- [x] README non-goal "no git repository setup and no CI" is stale now that the - repo is under git with CI -- [x] pre-commit hook not installed (`make hooks` once the target exists) - -Accepted divergences (no action): - -- flat single-package layout with `.go` files in the repo root — fine for a - small single-binary tool per the Go styleguide; the tracker audit agrees +- Flat single-package layout, with the `.go` files in the repo root. That is + fine for a small single-binary tool per the Go styleguide, and the tracker + audit agrees. diff --git a/cancel_test.go b/cancel_test.go index 20e72e2..577c54d 100644 --- a/cancel_test.go +++ b/cancel_test.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -168,8 +169,8 @@ func assertRecordsIntact(t *testing.T, db *sql.DB, before []string) { // TestSyncScanCancelledMidWalkKeepsRecords is the regression net under // the post-walk guard. The scan is cancelled part-way through the -// walk, so it reaches the guard holding a genuinely partial size -// census and a still-populated index of records the walk never got to. +// walk, so it reaches the guard having walked a genuinely partial tree +// and holding a still-populated index of records the walk never got to. // Every one of those records would look vanished to the update phase. // The guard is what stops the scan there, and this test is what // notices if it stops doing so: deleting the guard, or making it @@ -208,10 +209,10 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) { } // assertWalkGuardAborted checks that the scan stopped at the post-walk -// guard: with a census that is neither empty (the walk really ran) -// nor complete (it really was cut short), and with no record counted -// for removal. A removal count means the partial census was carried -// past the guard into the update phase, which is the failure this test +// guard: with a walk that reached some files (it really ran) but not +// all of them (it really was cut short), and with no record counted for +// removal. A removal count means the partial walk was carried past +// the guard into the update phase, which is the failure this test // exists to catch. func assertWalkGuardAborted(t *testing.T, st scanStats, err error) { t.Helper() @@ -222,30 +223,30 @@ func assertWalkGuardAborted(t *testing.T, st scanStats, err error) { } if st.unchanged == 0 { - t.Fatalf("stats = %+v: the census is empty, so the walk never "+ + t.Fatalf("stats = %+v: the walk reached no file, so it never "+ "ran and the guard was reached for the wrong reason", st) } if st.unchanged >= walkCancelFiles { - t.Fatalf("stats = %+v: the census covers the whole tree, so the "+ - "walk was not cut short", st) + t.Fatalf("stats = %+v: the walk reached the whole tree, so it "+ + "was not cut short", st) } // The workers drop every directory still queued once the scan is // cancelled, so only the files in the directories already in flight - // can add to the census after the fact. A census beyond that bound + // can still be reached after the fact. A walk beyond that bound // would mean the cancellation was not observed where it should have // been. limit := walkCancelAtDone + walkCancelInFlightFiles if st.unchanged > limit { - t.Errorf("census covers %d files, want at most %d: the walk kept "+ + t.Errorf("the walk reached %d files, want at most %d: it kept "+ "taking directories off the queue after cancellation", st.unchanged, limit) } if st.removed != 0 { t.Errorf("stats = %+v: the scan counted records for removal from "+ - "a partial census", st) + "a partial walk", st) } } @@ -762,7 +763,9 @@ func TestHashWorkerDropsQueuedRuns(t *testing.T) { var hashed atomic.Bool - hash := func(path string, size int64) (string, string, string, error) { + hash := func(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error, + ) { hashed.Store(true) return hashSignature(path, size) @@ -802,7 +805,9 @@ func TestHashWorkerAbandonsBlockedSend(t *testing.T) { // The scan is cancelled while the worker hashes, so the worker has // already passed the check that drops queued runs. - hash := func(path string, size int64) (string, string, string, error) { + hash := func(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error, + ) { cancel() return hashSignature(path, size) diff --git a/db.go b/db.go index de61db5..a27d163 100644 --- a/db.go +++ b/db.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -46,18 +47,20 @@ const dbFilePerm = 0o644 const lockFilePerm = 0o600 // createTableSQL is the schema applied to a fresh database. Paths are -// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime -// holds whole Unix seconds and mtime_nsec the nanoseconds within that -// second. +// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. +// disk_size holds the bytes the file takes on disk. mtime holds whole +// Unix seconds and mtime_nsec the nanoseconds within that second. head, +// tail and content each hold the raw 32 bytes of a SHA-256. const createTableSQL = ` CREATE TABLE files ( path BLOB PRIMARY KEY, size INTEGER NOT NULL, + disk_size INTEGER NOT NULL, mtime INTEGER NOT NULL, mtime_nsec INTEGER NOT NULL, - head TEXT NOT NULL, - tail TEXT NOT NULL, - content TEXT NOT NULL + head BLOB NOT NULL, + tail BLOB NOT NULL, + content BLOB NOT NULL ) WITHOUT ROWID ` @@ -67,14 +70,59 @@ const createIndexSQL = ` CREATE INDEX files_signature ON files (size, head, tail, content) ` +// createTotalsSQL creates the totals table, which holds one row: the +// number of records and the sizes of all records added up. Counting +// them over files would read every record. The three triggers below +// keep the row current, in the transaction of each change to files. +const createTotalsSQL = ` +CREATE TABLE totals ( + records INTEGER NOT NULL, + size INTEGER NOT NULL +) +` + +// insertTotalsSQL adds the one row of totals, for an empty files table. +const insertTotalsSQL = ` +INSERT INTO totals (records, size) VALUES (0, 0) +` + +// totalsAfterInsertSQL counts each inserted record into totals. +const totalsAfterInsertSQL = ` +CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN + UPDATE totals SET records = records + 1, size = size + NEW.size; +END +` + +// totalsAfterDeleteSQL takes each deleted record out of totals. +const totalsAfterDeleteSQL = ` +CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN + UPDATE totals SET records = records - 1, size = size - OLD.size; +END +` + +// totalsAfterUpdateSQL replaces a record's old size in totals with its +// new one. scan's upsert of an existing record is such an update. +const totalsAfterUpdateSQL = ` +CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN + UPDATE totals SET size = size + NEW.size - OLD.size; +END +` + +// recordCountSQL reads the number of records from totals. +const recordCountSQL = ` +SELECT records FROM totals +` + // upsertSQL inserts one file record, replacing any existing record for // the same path. const upsertSQL = ` -INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content) -VALUES (?, ?, ?, ?, ?, ?, ?) +INSERT INTO files ( + path, size, disk_size, mtime, mtime_nsec, head, tail, content +) +VALUES (?, ?, ?, ?, ?, ?, ?, ?) ON CONFLICT (path) DO UPDATE SET - size = excluded.size, mtime = excluded.mtime, - mtime_nsec = excluded.mtime_nsec, + size = excluded.size, disk_size = excluded.disk_size, + mtime = excluded.mtime, mtime_nsec = excluded.mtime_nsec, head = excluded.head, tail = excluded.tail, content = excluded.content ` @@ -92,35 +140,50 @@ var errSchemaVersion = errors.New("unsupported database schema version") // database. var errScanRunning = errors.New("another scan is running") +// errHashSize reports a record whose head, tail or content is not the +// 32 bytes of a SHA-256. +var errHashSize = errors.New("hash is not 32 bytes") + // databasePath resolves the database location: SFDUPES_DATABASE when // set and non-empty, the compiled-in default otherwise. It returns the -// path, with the working directory, its symlinks resolved, put in front -// of it when it is relative, and where the path came from, "default" or -// "from SFDUPES_DATABASE", for the line each subcommand prints first on -// stderr. +// path, made absolute by absolutePath, and where the path came from, +// "default" or "from SFDUPES_DATABASE", for the line each subcommand +// prints first on stderr. func databasePath() (string, string, error) { path, source := defaultDatabasePath, "default" if p := os.Getenv(databaseEnv); p != "" { path, source = p, "from "+databaseEnv } - if filepath.IsAbs(path) { - return path, source, nil - } - - // The working directory as the kernel reports it, with no symlink - // in it, as README "Database" says. os.Getwd may return PWD - // instead, which can name it through a symlink. - wd, err := unix.Getwd() + abs, err := absolutePath(path) if err != nil { return "", "", fmt.Errorf("resolve database path %s: %w", path, err) } - // Not filepath.Abs or filepath.Join: they clean the path, which - // drops each ".." together with the directory name before it. When - // that directory is a symlink, the cleaned path names another file. + return abs, source, nil +} + +// absolutePath returns path unchanged when it is absolute, and +// otherwise the working directory, a "/" and path. The working +// directory is the one the kernel reports, with no symlink in it; +// os.Getwd may return PWD instead, which can name it through a symlink. +// The result is not cleaned, as filepath.Abs and filepath.Join would +// clean it. Cleaning drops each ".." together with the directory name +// before it. When that name is a symlink, the cleaned path names +// another file than the one the system reaches by following the symlink +// first. +func absolutePath(path string) (string, error) { + if filepath.IsAbs(path) { + return path, nil + } + + wd, err := unix.Getwd() + if err != nil { + return "", err + } + // The working directory ends in "/" only when it is the root. - return strings.TrimSuffix(wd, "/") + "/" + path, source, nil + return strings.TrimSuffix(wd, "/") + "/" + path, nil } // createDatabaseDirectory creates the directory the database at path @@ -144,9 +207,14 @@ func createDatabaseDirectory(path string) error { // scanParams are the connection parameters for scan: read-write, with // WAL journaling and a busy timeout, so a report can run while a cron // scan is in progress. closeScanDatabase leaves WAL mode again. +// +// Each write to files fires a trigger, so SQLite keeps a statement +// journal for it. temp_store(MEMORY) keeps that journal in memory, +// which costs far less than a temporary file. const scanParams = "_pragma=busy_timeout(10000)" + "&_pragma=journal_mode(WAL)" + - "&_pragma=synchronous(NORMAL)" + "&_pragma=synchronous(NORMAL)" + + "&_pragma=temp_store(MEMORY)" // reportParams are the connection parameters for report, trees and // info: read-only, with the same busy timeout. They set no journal @@ -420,14 +488,14 @@ func createSchema(ctx context.Context, db *sql.DB) error { defer func() { _ = tx.Rollback() }() - _, err = tx.ExecContext(ctx, createTableSQL) - if err != nil { - return fmt.Errorf("create schema: %w", err) - } - - _, err = tx.ExecContext(ctx, createIndexSQL) - if err != nil { - return fmt.Errorf("create schema: %w", err) + for _, stmt := range []string{ + createTableSQL, createIndexSQL, createTotalsSQL, insertTotalsSQL, + totalsAfterInsertSQL, totalsAfterDeleteSQL, totalsAfterUpdateSQL, + } { + _, err = tx.ExecContext(ctx, stmt) + if err != nil { + return fmt.Errorf("create schema: %w", err) + } } _, err = tx.ExecContext(ctx, @@ -474,7 +542,7 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64), var n int64 - err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n) + err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n) if err != nil { return fmt.Errorf("count records: %w", err) } @@ -482,8 +550,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64), total(n) rows, err := tx.QueryContext(ctx, - "SELECT path, size, mtime, mtime_nsec, head, tail, content "+ - "FROM files ORDER BY path") + "SELECT path, size, disk_size, mtime, mtime_nsec, head, tail, "+ + "content FROM files ORDER BY path") if err != nil { return fmt.Errorf("read records: %w", err) } @@ -492,19 +560,25 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64), for rows.Next() { var ( - path []byte - sec, nsec int64 - r scanRec + path, head, tail, content []byte + sec, nsec int64 + r scanRec ) - err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail, - &r.content) + err = rows.Scan(&path, &r.size, &r.diskSize, &sec, &nsec, + &head, &tail, &content) if err != nil { return fmt.Errorf("read record: %w", err) } r.path = string(path) r.mtime = time.Unix(sec, nsec) + + err = r.setHashes(head, tail, content) + if err != nil { + return fmt.Errorf("read record %s: %w", escapePath(r.path), err) + } + fn(r) } @@ -527,10 +601,11 @@ ORDER BY size DESC, head DESC, tail DESC, content DESC ` // loadSignatureRows gives total the number of records, then streams -// every record's signature and path to fn, without its mtime, in the -// order of signatureRowsSQL. The count and the records are read in one -// transaction, so they agree while a scan is committing. An error from -// fn stops the reading and is returned as it is. +// every record's signature and path to fn, without its mtime or its +// disk_size, in the order of signatureRowsSQL. The count and the +// records are read in one transaction, so they agree while a scan is +// committing. An error from fn stops the reading and is returned as it +// is. func loadSignatureRows(ctx context.Context, db *sql.DB, total func(n int64), fn func(r scanRec) error, ) error { @@ -545,7 +620,7 @@ func loadSignatureRows(ctx context.Context, db *sql.DB, var n int64 - err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n) + err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n) if err != nil { return fmt.Errorf("count records: %w", err) } @@ -561,17 +636,22 @@ func loadSignatureRows(ctx context.Context, db *sql.DB, for rows.Next() { var ( - path []byte - r scanRec + path, head, tail, content []byte + r scanRec ) - err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path) + err = rows.Scan(&r.size, &head, &tail, &content, &path) if err != nil { return fmt.Errorf("read record: %w", err) } r.path = string(path) + err = r.setHashes(head, tail, content) + if err != nil { + return fmt.Errorf("read record %s: %w", escapePath(r.path), err) + } + err = fn(r) if err != nil { return err @@ -586,11 +666,27 @@ func loadSignatureRows(ctx context.Context, db *sql.DB, return nil } +// setHashes sets r's head, tail and content to the values read from +// those columns. Each must be the raw 32 bytes of a SHA-256. +func (r *scanRec) setHashes(head, tail, content []byte) error { + for _, b := range [][]byte{head, tail, content} { + if len(b) != sha256.Size { + return fmt.Errorf("%w: found %d bytes", errHashSize, len(b)) + } + } + + r.head = [sha256.Size]byte(head) + r.tail = [sha256.Size]byte(tail) + r.content = [sha256.Size]byte(content) + + return nil +} + // countRecords returns the number of records in the database. func countRecords(ctx context.Context, db *sql.DB) (int64, error) { var n int64 - err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n) + err := db.QueryRowContext(ctx, recordCountSQL).Scan(&n) if err != nil { return 0, fmt.Errorf("count records: %w", err) } @@ -598,16 +694,10 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) { return n, nil } -// databaseCountsSQL counts the records, those with head and tail -// hashes and those with a content hash, and adds up their sizes, in one -// pass over the table. A SUM over no rows is NULL, so on an empty table -// each SUM falls back to 0. +// databaseCountsSQL reads the number of records and the sizes of all +// records added up from totals. const databaseCountsSQL = ` -SELECT COUNT(*), - COALESCE(SUM(head <> '' AND tail <> ''), 0), - COALESCE(SUM(content <> ''), 0), - COALESCE(SUM(size), 0) -FROM files +SELECT records, size FROM totals ` // databaseCounts is what info reads from the database: its schema @@ -615,8 +705,6 @@ FROM files type databaseCounts struct { schemaVersion int records int64 // every record - headTail int64 // records with head and tail hashes - content int64 // records with a content hash size int64 // the sizes of all records added up } @@ -644,7 +732,7 @@ func readDatabaseCounts(ctx context.Context, } err = tx.QueryRowContext(ctx, databaseCountsSQL). - Scan(&c.records, &c.headTail, &c.content, &c.size) + Scan(&c.records, &c.size) if err != nil { return c, fmt.Errorf("count records: %w", err) } @@ -652,90 +740,15 @@ func readDatabaseCounts(ctx context.Context, return c, nil } -// loadFileMeta streams every record's path, size, mtime, and whether -// it carries hashes to fn. Scan change detection needs no hash -// values, and skipping the hash columns keeps the scan's in-memory -// index small on multi-million-file databases. +// loadFileMeta streams every record's path, size, disk_size and mtime +// to fn. Scan change detection needs no hash values, and skipping the +// hash columns keeps the scan's in-memory index small on +// multi-million-file databases. func loadFileMeta(ctx context.Context, db *sql.DB, - fn func(path string, size int64, mtime time.Time, hashed bool), + fn func(path string, m fileMeta), ) error { rows, err := db.QueryContext(ctx, - "SELECT path, size, mtime, mtime_nsec, head <> '' FROM files") - if err != nil { - return fmt.Errorf("read records: %w", err) - } - - defer func() { _ = rows.Close() }() - - for rows.Next() { - var ( - path []byte - size, sec, nsec int64 - hashed int64 - ) - - err = rows.Scan(&path, &size, &sec, &nsec, &hashed) - if err != nil { - return fmt.Errorf("read record: %w", err) - } - - fn(string(path), size, time.Unix(sec, nsec), hashed != 0) - } - - err = rows.Err() - if err != nil { - return fmt.Errorf("read records: %w", err) - } - - return nil -} - -// contentGroupsSQL selects each group (the records sharing a size, -// head, and tail) of records of at least headTailMin bytes that has more -// than one record and at least one record without a content hash, with -// the number of its records. -const contentGroupsSQL = ` -SELECT size, head, tail, COUNT(*) AS records -FROM files -WHERE size >= ? AND head <> '' -GROUP BY size, head, tail -HAVING COUNT(*) > 1 AND SUM(content = '') > 0 -` - -// contentCandidatesSQL selects every record of the groups -// contentGroupsSQL selects, with whether each record has a content -// hash. SQLite does the grouping, so no other record's hashes are -// loaded into memory; the rows come ordered by size, head, and tail, so -// each group's rows arrive together. -const contentCandidatesSQL = ` -SELECT f.path, f.size, f.mtime, f.mtime_nsec, f.head, f.tail, f.content <> '' -FROM files AS f -JOIN (` + contentGroupsSQL + `) AS g USING (size, head, tail) -ORDER BY size, head, tail -` - -// countContentCandidates returns the number of rows of -// contentCandidatesSQL, adding up the records of the groups -// contentGroupsSQL selects, so it neither joins nor sorts. -func countContentCandidates(ctx context.Context, db *sql.DB) (int64, error) { - var n int64 - - err := db.QueryRowContext(ctx, - "SELECT COALESCE(SUM(records), 0) FROM ("+contentGroupsSQL+")", - headTailMin).Scan(&n) - if err != nil { - return 0, fmt.Errorf("count content candidates: %w", err) - } - - return n, nil -} - -// loadContentCandidates streams the rows of contentCandidatesSQL to fn: -// each record, without its content hash, and whether it has one. -func loadContentCandidates(ctx context.Context, db *sql.DB, - fn func(r scanRec, hashed bool), -) error { - rows, err := db.QueryContext(ctx, contentCandidatesSQL, headTailMin) + "SELECT path, size, disk_size, mtime, mtime_nsec FROM files") if err != nil { return fmt.Errorf("read records: %w", err) } @@ -745,20 +758,18 @@ func loadContentCandidates(ctx context.Context, db *sql.DB, for rows.Next() { var ( path []byte + m fileMeta sec, nsec int64 - r scanRec - hashed int64 ) - err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail, - &hashed) + err = rows.Scan(&path, &m.size, &m.diskSize, &sec, &nsec) if err != nil { return fmt.Errorf("read record: %w", err) } - r.path = string(path) - r.mtime = time.Unix(sec, nsec) - fn(r, hashed != 0) + m.mtime = time.Unix(sec, nsec) + + fn(string(path), m) } err = rows.Err() @@ -776,21 +787,21 @@ func loadContentCandidates(ctx context.Context, db *sql.DB, // WAL small and let concurrent reports observe progress. const updateBatchSize = 10000 -// applyChanges writes one scan's database changes — upserts for new and -// changed files, deletes for vanished ones — in batched transactions. -// Progress is rendered on prog (one increment per change). +// applyChanges writes upserts for new and changed files and deletes for +// vanished ones, in batched transactions. Progress is rendered on prog +// (one increment per change). func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec, deletes []string, prog *progress, ) error { for batch := range slices.Chunk(upserts, updateBatchSize) { - err := applyBatch(ctx, db, batch, nil, prog) + err := applyBatch(ctx, db, batch, nil, nil, prog) if err != nil { return err } } for batch := range slices.Chunk(deletes, updateBatchSize) { - err := applyBatch(ctx, db, nil, batch, prog) + err := applyBatch(ctx, db, nil, nil, batch, prog) if err != nil { return err } @@ -799,10 +810,26 @@ func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec, return nil } -// applyBatch commits one batch of upserts and deletes in a single -// transaction. +// applyDiskSizes writes the disk_size of each of recs into its record, +// in batched transactions, and leaves the rest of the record alone. +// Progress is rendered on prog (one increment per record). +func applyDiskSizes(ctx context.Context, db *sql.DB, recs []fileRec, + prog *progress, +) error { + for batch := range slices.Chunk(recs, updateBatchSize) { + err := applyBatch(ctx, db, nil, batch, nil, prog) + if err != nil { + return err + } + } + + return nil +} + +// applyBatch commits one batch of upserts, disk_size updates and +// deletes in a single transaction. func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec, - deletes []string, prog *progress, + diskSizes []fileRec, deletes []string, prog *progress, ) error { tx, err := db.BeginTx(ctx, nil) if err != nil { @@ -816,6 +843,11 @@ func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec, return err } + err = execDiskSizes(ctx, tx, diskSizes, prog) + if err != nil { + return err + } + err = execDeletes(ctx, tx, deletes, prog) if err != nil { return err @@ -841,8 +873,9 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec, defer func() { _ = st.Close() }() for _, r := range upserts { - _, err = st.ExecContext(ctx, []byte(r.path), r.size, - r.mtime.Unix(), r.mtime.Nanosecond(), r.head, r.tail, r.content) + _, err = st.ExecContext(ctx, []byte(r.path), r.size, r.diskSize, + r.mtime.Unix(), r.mtime.Nanosecond(), + r.head[:], r.tail[:], r.content[:]) if err != nil { return fmt.Errorf("upsert %s: %w", r.path, err) } @@ -853,6 +886,31 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec, return nil } +// execDiskSizes sets the disk_size of the record of each of recs. It +// does not touch size, so the totals triggers do not fire. +func execDiskSizes(ctx context.Context, tx *sql.Tx, recs []fileRec, + prog *progress, +) error { + st, err := tx.PrepareContext(ctx, + "UPDATE files SET disk_size = ? WHERE path = ?") + if err != nil { + return fmt.Errorf("prepare disk_size update: %w", err) + } + + defer func() { _ = st.Close() }() + + for _, r := range recs { + _, err = st.ExecContext(ctx, r.diskSize, []byte(r.path)) + if err != nil { + return fmt.Errorf("update disk_size of %s: %w", r.path, err) + } + + prog.increment() + } + + return nil +} + // execDeletes removes the records for paths no longer present. func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string, prog *progress, diff --git a/db_test.go b/db_test.go index 427fb61..6d9da76 100644 --- a/db_test.go +++ b/db_test.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -473,17 +474,73 @@ func TestLoadRowsCountAgreeDuringScan(t *testing.T) { } } -// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL -// from the files_signature index alone, with nothing to sort: the -// first record comes at once, so report's read phase counts each -// record as it comes rather than waiting on SQLite for all of them. -func TestSignatureRowsNeedNoSort(t *testing.T) { +// TestLoadRowsRefuseWrongSizeHash stores a record whose head, tail or +// content is not the 32 bytes of a SHA-256: either one byte, or the 64 +// bytes of a SHA-256 written as hex text. loadFileRows and +// loadSignatureRows must each fail on it with an error naming its path. +func TestLoadRowsRefuseWrongSizeHash(t *testing.T) { t.Parallel() - db := openTestDB(t) + hexText := strings.Repeat("ab", sha256.Size) - rows, err := db.QueryContext(t.Context(), - "EXPLAIN QUERY PLAN "+signatureRowsSQL) + cases := []struct { + name string + update string + value any + }{ + {"one-byte head", "UPDATE files SET head = ?", []byte{0}}, + {"hex-text head", "UPDATE files SET head = ?", hexText}, + {"one-byte tail", "UPDATE files SET tail = ?", []byte{0}}, + {"hex-text tail", "UPDATE files SET tail = ?", hexText}, + {"one-byte content", "UPDATE files SET content = ?", []byte{0}}, + {"hex-text content", "UPDATE files SET content = ?", hexText}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + t.Parallel() + + db := openTestDB(t) + + err := applyChanges(t.Context(), db, + []scanRec{{path: "/a/f"}}, nil, nil) + if err != nil { + t.Fatal(err) + } + + _, err = db.ExecContext(t.Context(), c.update, c.value) + if err != nil { + t.Fatal(err) + } + + loads := map[string]func() error{ + "loadFileRows": func() error { + return loadFileRows(t.Context(), db, func(int64) {}, + func(scanRec) {}) + }, + "loadSignatureRows": func() error { + return loadSignatureRows(t.Context(), db, func(int64) {}, + func(scanRec) error { return nil }) + }, + } + + for name, load := range loads { + err := load() + if !errors.Is(err, errHashSize) || + !strings.Contains(err.Error(), "/a/f") { + t.Errorf("%s: err = %v, want %v naming /a/f", + name, err, errHashSize) + } + } + }) + } +} + +// queryPlan returns the detail of each line of SQLite's EXPLAIN QUERY +// PLAN for query. +func queryPlan(t *testing.T, db *sql.DB, query string) []string { + t.Helper() + + rows, err := db.QueryContext(t.Context(), "EXPLAIN QUERY PLAN "+query) if err != nil { t.Fatal(err) } @@ -511,12 +568,164 @@ func TestSignatureRowsNeedNoSort(t *testing.T) { t.Fatal(err) } + return plan +} + +// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL +// from the files_signature index alone, with nothing to sort: the +// first record comes at once, so report's read phase counts each +// record as it comes rather than waiting on SQLite for all of them. +func TestSignatureRowsNeedNoSort(t *testing.T) { + t.Parallel() + + db := openTestDB(t) + + plan := queryPlan(t, db, signatureRowsSQL) + want := []string{"SCAN files USING COVERING INDEX files_signature"} if !slices.Equal(plan, want) { t.Errorf("query plan = %q, want %q", plan, want) } } +// TestCountsReadOnlyTotals checks that SQLite answers the queries for +// the number of records and their total size from the one row of +// totals. A plan that reads files or its index reads every record. +func TestCountsReadOnlyTotals(t *testing.T) { + t.Parallel() + + db := openTestDB(t) + + for _, query := range []string{recordCountSQL, databaseCountsSQL} { + plan := queryPlan(t, db, query) + + want := []string{"SCAN totals"} + if !slices.Equal(plan, want) { + t.Errorf("query plan of %q = %q, want %q", query, plan, want) + } + } +} + +// TestCountsComeFromTotals sets the row of totals to numbers that files +// does not hold, then checks that scan's load phase, the read phases of +// trees and report, and info each report the numbers from totals. +func TestCountsComeFromTotals(t *testing.T) { + t.Parallel() + + // files is empty, so a count over files gives 0 records of 0 bytes. + const records, size = 7, 9000 + + db := openTestDB(t) + + _, err := db.ExecContext(t.Context(), + "UPDATE totals SET records = ?, size = ?", records, size) + if err != nil { + t.Fatal(err) + } + + scanTotal, err := countRecords(t.Context(), db) + if err != nil { + t.Fatal(err) + } + + var treesTotal, reportTotal int64 + + err = loadFileRows(t.Context(), db, + func(n int64) { treesTotal = n }, func(scanRec) {}) + if err != nil { + t.Fatal(err) + } + + err = loadSignatureRows(t.Context(), db, + func(n int64) { reportTotal = n }, + func(scanRec) error { return nil }) + if err != nil { + t.Fatal(err) + } + + info, err := readDatabaseCounts(t.Context(), db) + if err != nil { + t.Fatal(err) + } + + for name, got := range map[string]int64{ + "scan's load": scanTotal, + "trees' read": treesTotal, + "report's read": reportTotal, + "info": info.records, + } { + if got != records { + t.Errorf("%s: %d records, want %d from totals", + name, got, records) + } + } + + if info.size != size { + t.Errorf("info: size %d, want %d from totals", info.size, size) + } +} + +// assertTotalsMatchFiles fails unless the row of totals holds the +// number of records and the sizes of all records added up, as counted +// over files. +func assertTotalsMatchFiles(t *testing.T, db *sql.DB) { + t.Helper() + + got, err := readDatabaseCounts(t.Context(), db) + if err != nil { + t.Fatal(err) + } + + var records, size int64 + + err = db.QueryRowContext(t.Context(), + "SELECT COUNT(*), COALESCE(SUM(size), 0) FROM files"). + Scan(&records, &size) + if err != nil { + t.Fatal(err) + } + + if got.records != records || got.size != size { + t.Errorf("totals = %d records of %d bytes, files holds %d of %d", + got.records, got.size, records, size) + } +} + +// TestTotalsMatchFiles scans the smoke tree, rescans it after two files +// change size, and rescans it after two files are removed. After each +// scan the row of totals must agree with files. +func TestTotalsMatchFiles(t *testing.T) { + t.Parallel() + + dir := buildSmokeTree(t) + db := openTestDB(t) + + syncTree(t, db, dir) + assertTotalsMatchFiles(t, db) + + writeFile(t, dir, "a/one.bin", pattern(10, 2500)) + writeFile(t, dir, "t1/f1", pattern(30, 1000)) + + if st := syncTree(t, db, dir); st.updated != 2 { + t.Fatalf("rescan stats = %+v, want 2 updated", st) + } + + assertTotalsMatchFiles(t, db) + + for _, name := range []string{"a/unique.bin", "t2/sub/f2"} { + err := os.Remove(filepath.Join(dir, name)) + if err != nil { + t.Fatal(err) + } + } + + if st := syncTree(t, db, dir); st.removed != 2 { + t.Fatalf("rescan stats = %+v, want 2 removed", st) + } + + assertTotalsMatchFiles(t, db) +} + func TestApplyChangesRoundTrip(t *testing.T) { t.Parallel() @@ -527,12 +736,13 @@ func TestApplyChangesRoundTrip(t *testing.T) { // written. recs := []scanRec{ { - size: 2, mtime: time.Unix(20, 999_999_999), head: "h2", tail: "t2", - content: "c2", path: "/a/tab\tnew\nline", + size: 2, mtime: time.Unix(20, 999_999_999), head: testHash("h2"), + tail: testHash("t2"), content: testHash("c2"), + path: "/a/tab\tnew\nline", }, { - size: 1, mtime: time.Unix(10, 0), head: "h1", tail: "t1", - content: "c1", path: "/a/x", + size: 1, mtime: time.Unix(10, 0), head: testHash("h1"), + tail: testHash("t1"), content: testHash("c1"), path: "/a/x", }, } @@ -553,8 +763,8 @@ func TestApplyChangesRoundTrip(t *testing.T) { // An upsert for an existing path updates in place; a delete // removes exactly its path. upd := scanRec{ - size: 3, mtime: time.Unix(30, 0), head: "h3", tail: "t3", content: "c3", - path: "/a/x", + size: 3, mtime: time.Unix(30, 0), head: testHash("h3"), + tail: testHash("t3"), content: testHash("c3"), path: "/a/x", } err = applyChanges(t.Context(), db, []scanRec{upd}, @@ -581,8 +791,8 @@ func TestApplyChangesBatching(t *testing.T) { recs := make([]scanRec, 0, n) for i := range n { recs = append(recs, scanRec{ - size: int64(i), mtime: time.Unix(1, 0), head: "h", tail: "t", - path: fmt.Sprintf("/batch/%07d", i), + size: int64(i), mtime: time.Unix(1, 0), head: testHash("h"), + tail: testHash("t"), path: fmt.Sprintf("/batch/%07d", i), }) } @@ -612,62 +822,3 @@ func TestApplyChangesBatching(t *testing.T) { t.Fatalf("records = %d, want 0", len(got)) } } - -// TestCountContentCandidates checks that countContentCandidates counts -// the records the content phase's search returns: those of each group -// of two or more records of headTailMin or more with a head, where at -// least one has no content hash. -func TestCountContentCandidates(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - - // Each group's records: size, head, and content, one per record. - groups := []struct { - size int64 - head string - contents []string - }{ - {headTailMin, "h", []string{"", "c", "c"}}, // returned - {headTailMin + 1, "h", []string{"", ""}}, // returned - {headTailMin + 2, "h", []string{"c", "c"}}, // all hashed - {headTailMin + 3, "h", []string{""}}, // one record - {headTailMin - 1, "h", []string{"", ""}}, // too small - {headTailMin + 4, "", []string{"", ""}}, // no head - } - - var recs []scanRec - - for _, g := range groups { - for _, content := range g.contents { - recs = append(recs, scanRec{ - size: g.size, mtime: time.Unix(1, 0), head: g.head, - tail: "t", content: content, - path: fmt.Sprintf("/r/%d", len(recs)), - }) - } - } - - err := applyChanges(t.Context(), db, recs, nil, nil) - if err != nil { - t.Fatalf("applyChanges: %v", err) - } - - got, err := countContentCandidates(t.Context(), db) - if err != nil { - t.Fatalf("countContentCandidates: %v", err) - } - - var returned int64 - - err = loadContentCandidates(t.Context(), db, - func(scanRec, bool) { returned++ }) - if err != nil { - t.Fatalf("loadContentCandidates: %v", err) - } - - if got != 5 || returned != 5 { - t.Errorf("count = %d, records returned = %d, want 5 and 5", - got, returned) - } -} diff --git a/dwim.go b/dwim.go new file mode 100644 index 0000000..d195f16 --- /dev/null +++ b/dwim.go @@ -0,0 +1,37 @@ +package main + +import ( + "context" + "fmt" + "io" + "os" + "runtime" +) + +// runDwim implements the dwim subcommand: it runs what "sfdupes scan +// root" runs with no flags, then what "sfdupes trees" runs, and ends +// with a line on stderr saying how to print the report again. The dwim +// command passes "/" as root. It catches SIGINT and SIGTERM as scan +// does, but only during the scan. An error from the scan, an interrupt +// included, ends it before the report. During the report, a signal +// ends it at once, as it ends trees. +func runDwim(ctx context.Context, root string, stdout io.Writer) error { + scanCtx, stop := interruptContext(ctx) + err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false) + + stop() + + if err != nil { + return err + } + + err = runTrees(ctx, stdout) + if err != nil { + return err + } + + fmt.Fprintln(os.Stderr, + `dwim: run "sfdupes trees" to print this report again`) + + return nil +} diff --git a/dwim_test.go b/dwim_test.go new file mode 100644 index 0000000..5622ad9 --- /dev/null +++ b/dwim_test.go @@ -0,0 +1,83 @@ +package main + +import ( + "bytes" + "path/filepath" + "strings" + "testing" +) + +func TestRunDwimPrintsLargestTreesFirst(t *testing.T) { + // dwim scans its root, then prints the trees report, the larger of + // two duplicate-tree pairs first, and ends by saying how to print + // the report again. + t.Setenv(databaseEnv, testDBPath(t)) + + root := realTempDir(t) + + small := []string{ + filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))), + filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))), + } + large := []string{ + filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))), + filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))), + } + + var stdout bytes.Buffer + + stderr := captureStderr(t) + + err := runDwim(t.Context(), root, &stdout) + if err != nil { + t.Fatalf("runDwim: %v; stderr: %s", err, stderr()) + } + + lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n") + if len(lines) != 3 { + t.Fatalf("stdout = %q, want a header and two groups", + stdout.String()) + } + + wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t" + if !strings.HasPrefix(lines[1], wantFirst) { + t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst) + } + + wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t" + if !strings.HasPrefix(lines[2], wantSecond) { + t.Errorf("second group = %q, want it to start %q", + lines[2], wantSecond) + } + + last := "\ndwim: run \"sfdupes trees\" to print this report again\n" + if got := stderr(); !strings.HasSuffix(got, last) { + t.Errorf("stderr = %q, want it to end %q", got, last) + } +} + +func TestRunDwimScanErrorPrintsNoReport(t *testing.T) { + // A fatal scan error ends dwim before the report: a root that does + // not exist fails the scan as it fails scan itself. + t.Setenv(databaseEnv, testDBPath(t)) + + missing := filepath.Join(realTempDir(t), "missing") + + var stdout bytes.Buffer + + stderr := captureStderr(t) + + err := runDwim(t.Context(), missing, &stdout) + if err == nil { + t.Fatal("runDwim on a missing root: error = nil, want it fatal") + } + + if got := stdout.String(); got != "" { + t.Errorf("stdout = %q, want no report", got) + } + + if got := stderr(); strings.Contains(got, "trees:") || + strings.Contains(got, "dwim:") { + t.Errorf("stderr = %q, want nothing from trees or dwim", got) + } +} diff --git a/exclude_test.go b/exclude_test.go index 20f195d..25b593e 100644 --- a/exclude_test.go +++ b/exclude_test.go @@ -17,7 +17,9 @@ import ( func TestDefaultExcludes(t *testing.T) { t.Parallel() - // README "Rules for the walk": each system's list. + // README "Rules for the walk": the Linux and macOS lists, and none + // on any other system. The Windows list exists in the code only: the + // binary does not build for Windows, so the README leaves it out. cases := map[string][]string{ goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}, goosDarwin: { @@ -164,9 +166,8 @@ func TestDefaultExcludesHelp(t *testing.T) { // TestScanDoesNotEnterDefaultExclude checks that the walk does not // enter a default exclude below an operand. It says so on a line of its // own and does not count it as skipped. The records below it count as -// outside the operands: the record of a file gone from it is kept, and -// its size joins the size census, while a file added to it is not -// recorded. +// outside the operands: the record of a file gone from it is kept, +// while a file added to it is not recorded. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestScanDoesNotEnterDefaultExclude(t *testing.T) { @@ -186,10 +187,6 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) { unseen := writeFile(t, excluded, "new", pattern(2, 10)) - // The same size as gone, so it is hashed only if the census holds the - // size of gone's record. - added := writeFile(t, dir, "added", pattern(3, 100)) - st, err := syncScan(t.Context(), db, []string{dir}, 1, false, walkRules{excludes: []string{excluded}}) if err != nil { @@ -203,11 +200,6 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) { recs := dbRecords(t, db) recordByPath(t, recs, gone) - if recordByPath(t, recs, added).head == "" { - t.Errorf("%s not hashed: the census lacks the size of %s", - added, gone) - } - if slices.Contains(recordPaths(recs), unseen) { t.Errorf("%s recorded: the walk entered %s", unseen, excluded) } @@ -294,7 +286,7 @@ func TestRunScanTempDirOperand(t *testing.T) { path := testDBPath(t) t.Setenv(databaseEnv, path) - dir := t.TempDir() + dir := realTempDir(t) if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) { t.Skip(dir + " is not inside a default exclude") } @@ -391,9 +383,8 @@ func TestRunScanSkipsNonPersistentOperand(t *testing.T) { // TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem // that is not persistent beside one on the tests' temporary directory. // The records an earlier scan left in it count as outside the operands: -// the record of a file gone from it is kept, and its size joins the size -// census, while a file added to it is not recorded, and its size does -// not join the census. +// the record of a file gone from it is kept, while a file added to it +// is not recorded. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestScanKeepsRecordsOnNonPersistent(t *testing.T) { @@ -413,11 +404,6 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) { unseen := writeFile(t, shm, "new", pattern(2, 10)) - // added has the size of gone, and lonely that of unseen: each is - // hashed only if the census holds that size. - added := writeFile(t, dir, "added", pattern(3, 100)) - lonely := writeFile(t, dir, "lonely", pattern(4, 10)) - st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules) if err != nil { t.Fatal(err) @@ -430,15 +416,6 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) { recs := dbRecords(t, db) recordByPath(t, recs, gone) - if recordByPath(t, recs, added).head == "" { - t.Errorf("%s not hashed: the census lacks the size of %s", - added, gone) - } - - if recordByPath(t, recs, lonely).head != "" { - t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen) - } - if slices.Contains(recordPaths(recs), unseen) { t.Errorf("%s recorded", unseen) } @@ -450,51 +427,6 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) { } } -// TestScanContentSkipsNonPersistent checks that the content phase does -// not read a file on a filesystem that is not persistent, though an -// earlier scan left a record for it: it is treated as a changed file, -// so its record is left as it is and its identical copy in another -// tree, having no other match, is not read either. -func TestScanContentSkipsNonPersistent(t *testing.T) { - t.Parallel() - - shm, _, rules := shmTempDir(t) - db := openTestDB(t) - - // A scan that, without the list of filesystems, records a with a - // head and tail but no content hash. - a := sparseFileWithoutMatch(t, shm, "a", headTailMin) - - syncTree(t, db, shm) - - before := recordByPath(t, dbRecords(t, db), a) - if before.head == "" || before.content != "" { - t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before) - } - - b := sparseFile(t, t.TempDir(), "b", headTailMin) - - st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false, - rules) - if err != nil { - t.Fatal(err) - } - - if st.skipped != 0 { - t.Errorf("skipped = %d, want 0", st.skipped) - } - - recs := dbRecords(t, db) - if r := recordByPath(t, recs, a); r != before { - t.Errorf("record = %+v, want it left as %+v", r, before) - } - - if r := recordByPath(t, recs, b); r.content != "" { - t.Errorf("%s: content = %q, want none: its only match is on %s", - b, r.content, shm) - } -} - // TestWalkLeavesOutNonPersistent starts the walk itself in a directory // on a filesystem that is not persistent, which a scan never does: it // drops such an operand first. Below a directory on a persistent diff --git a/info.go b/info.go index 4f705c4..c30cac8 100644 --- a/info.go +++ b/info.go @@ -26,11 +26,11 @@ func infoPhase(name string) string { // it. Any database problem, including a missing database, is fatal. // // The work runs in two phases, each under a progress display of its -// own on stderr: open (open the database) and count (count the records, -// with one query). Each display ends before the next phase starts, and -// before an error is returned to be printed. The details are written -// once the last display has ended, so no display is drawn while they -// are written. +// own on stderr: open (open the database) and count (read the number of +// records and their total size from the totals table). Each display +// ends before the next phase starts, and before an error is returned +// to be printed. The details are written once the last display has +// ended, so no display is drawn while they are written. func runInfo(ctx context.Context, stdout io.Writer) error { dbPath, source, err := databasePath() if err != nil { @@ -56,7 +56,7 @@ func runInfo(ctx context.Context, stdout io.Writer) error { defer func() { _ = db.Close() }() - // One query counts everything, so the phase never has a total. + // One row holds both counts, so the phase never has a total. prog = newProgress(progressParams{ label: infoPhase("count"), total: -1, waitingFor: countingRecords, @@ -113,9 +113,6 @@ func writeInfo(stdout io.Writer, dbPath, source string, dbFile.ModTime().UTC().Format(time.RFC3339)) fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion) fmt.Fprintf(&b, "records\t%d\n", counts.records) - fmt.Fprintf(&b, "records with head and tail hashes\t%d\n", - counts.headTail) - fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content) fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n", counts.size, humanBytes(counts.size)) diff --git a/info_test.go b/info_test.go index 1d308ed..5d6a58e 100644 --- a/info_test.go +++ b/info_test.go @@ -12,23 +12,25 @@ import ( // infoOpenLine is the plain line of info's open phase when it opens the // database at once; it prints it as the phase starts and as it ends. const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" // infoCountLine is the plain line of info's count phase when the count // takes under a second; it prints it as the phase starts and as it // ends. const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " + - "eta unknown" + "remain unknown" -// infoRecs are four records: a duplicate pair with every hash, a file -// of 20 MiB with head and tail hashes but no content hash yet, and a -// file never hashed. +// infoRecs are four records: a duplicate pair, a file of 20 MiB and +// another small file. func infoRecs() []scanRec { + h, tl, c, u := testHash("h"), testHash("t"), testHash("c"), testHash("u") + bigH, bigT, bigC := testHash("bh"), testHash("bt"), testHash("bc") + return []scanRec{ - {size: 5, head: "h", tail: "t", content: "c", path: "/d/a"}, - {size: 5, head: "h", tail: "t", content: "c", path: "/d/b"}, - {size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"}, - {size: 7, path: "/d/unique"}, + {size: 5, head: h, tail: tl, content: c, path: "/d/a"}, + {size: 5, head: h, tail: tl, content: c, path: "/d/b"}, + {size: 20 << 20, head: bigH, tail: bigT, content: bigC, path: "/d/big"}, + {size: 7, head: u, tail: u, content: u, path: "/d/unique"}, } } @@ -45,16 +47,12 @@ func TestRunInfo(t *testing.T) { "empty": { want: "schema version\t1\n" + "records\t0\n" + - "records with head and tail hashes\t0\n" + - "records with content hash\t0\n" + "size of recorded files\t0 (0 B)\n", }, "filled": { recs: infoRecs(), want: "schema version\t1\n" + "records\t4\n" + - "records with head and tail hashes\t3\n" + - "records with content hash\t2\n" + "size of recorded files\t20971537 (20.0 MiB)\n", }, } diff --git a/main.go b/main.go index 2178408..1d4fed2 100644 --- a/main.go +++ b/main.go @@ -1,14 +1,12 @@ // Command sfdupes quickly identifies candidate duplicate files across // very large filesystems without reading every byte of every file. // Files are considered duplicates when their sizes are equal and they -// agree on a short ladder of SHA-256 hashes. A file under 10 MiB is -// hashed in full. A larger file is compared on the hashes of its first -// and last 64 KiB, and only when those match another file's is its -// content hash computed and compared: of the whole file when it is -// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB -// or larger. scan maintains a persistent SQLite database of file -// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) -// that the other subcommands read. +// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64 +// KiB, and of the content. The content hash covers the whole file under +// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large +// file is never read in full. scan maintains a persistent SQLite +// database of file signatures (SFDUPES_DATABASE, default +// /var/lib/sfdupes/db.sqlite) that the other subcommands read. // // Usage: // @@ -16,6 +14,7 @@ // sfdupes report > dupes.tsv // sfdupes trees > dupetrees.tsv // sfdupes info +// sfdupes dwim // sfdupes --version // // See README.md for the complete specification. @@ -47,6 +46,7 @@ const ( cmdReport = "report" cmdTrees = "trees" cmdInfo = "info" + cmdDwim = "dwim" ) // errNoSubcommand is returned by the root command when it is invoked @@ -98,8 +98,8 @@ func main() { // exit code. It is the program's single exit point: the subcommands // return their errors instead of exiting, so every deferred cleanup — // above all closing the database, which checkpoints the SQLite WAL — -// runs before the process ends. The report, trees and info subcommands -// write their data to stdout. +// runs before the process ends. The report, trees, info and dwim +// subcommands write their data to stdout. func run(args []string, stdout, stderr io.Writer) int { // A nil slice makes cobra fall back to os.Args, which would let a // test binary's own flags reach the command tree. @@ -203,7 +203,16 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command { }), } - root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd) + dwimCmd := &cobra.Command{ + Use: cmdDwim, + Short: "Scan / with the default excludes, then print the duplicate-tree report", + Args: cobra.NoArgs, + RunE: runE(func(ctx context.Context, _ []string) error { + return runDwim(ctx, "/", stdout) + }), + } + + root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd) return root } @@ -238,7 +247,7 @@ func newScanCommand() *cobra.Command { }), } scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(), - "concurrent workers for the walk, hash, and content phases") + "concurrent workers for the walk and hash phases") scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false, "do not cross filesystem boundaries") scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes", diff --git a/main_test.go b/main_test.go index 8e39eab..e0abba1 100644 --- a/main_test.go +++ b/main_test.go @@ -14,6 +14,8 @@ import ( "strconv" "strings" "testing" + + "golang.org/x/sys/unix" ) // usageMarker is the first line of cobra's usage text, which a usage @@ -180,9 +182,9 @@ func capture(t *testing.T, std **os.File) func() string { } // brokenDatabase writes a database that opens cleanly and passes the -// schema-version check but has no files table, so the first query -// fails with the database already open: a fatal error on a path that -// owns an open database. It closes the database the way scan does. +// schema-version check but has no tables, so the first query fails +// with the database already open: a fatal error on a path that owns an +// open database. It closes the database the way scan does. func brokenDatabase(t *testing.T) string { t.Helper() @@ -265,8 +267,9 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) { assertFatalOutput(t, stderr.String(), stdout()) // Proof that the failure happened after the open: only a - // query against the opened database can report this. - if !strings.Contains(stderr.String(), "no such table: files") { + // query against the opened database can report this. Each + // subcommand's first query reads totals. + if !strings.Contains(stderr.String(), "no such table: totals") { t.Errorf("stderr = %q, want the failure to come from a "+ "query on the open database", stderr.String()) } @@ -393,7 +396,7 @@ func TestRunNamesRelativeDatabase(t *testing.T) { } func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) { - // README §Database: the path names the file exactly. With link a + // README §Settings: the path names the file exactly. With link a // symlink to the directory real/sub, link/../db.sqlite names // real/db.sqlite, not db.sqlite beside link. Every subcommand must // use that file, relative or absolute, and its first line must name @@ -619,6 +622,7 @@ func TestRunUsageErrors(t *testing.T) { "report with args": {[]string{cmdReport, "x"}, usageMarker}, "trees with args": {[]string{cmdTrees, "x"}, usageMarker}, "info with args": {[]string{cmdInfo, "x"}, usageMarker}, + "dwim with args": {[]string{cmdDwim, "x"}, usageMarker}, "unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker}, "unknown subcommand": {[]string{"nope"}, "unknown command"}, } @@ -648,7 +652,7 @@ func TestRunUsageErrors(t *testing.T) { } func TestRunScanRejectsWorkersBelowOne(t *testing.T) { - // README §scan mode: --workers below 1 is a usage error reported in + // README §Settings: --workers below 1 is a usage error reported in // one line on stderr, before the scan opens the database. for _, workers := range []string{"0", "-1"} { t.Run(workers, func(t *testing.T) { @@ -686,8 +690,8 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) { func TestRunHelp(t *testing.T) { t.Parallel() - // README §Subcommands: help goes to stderr, exits 0, and leaves - // stdout empty. + // README §Settings: help goes to stderr, exits 0, and leaves stdout + // empty. cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}} for _, args := range cases { @@ -713,8 +717,8 @@ func TestRunHelp(t *testing.T) { func TestRunVersion(t *testing.T) { t.Parallel() - // README §Subcommands: the version is one line on stdout, with - // nothing on stderr, and exits 0. + // README §Settings: the version is one line on stdout, with nothing + // on stderr, and exits 0. info, _ := debug.ReadBuildInfo() for _, arg := range []string{"--version", "-v"} { @@ -773,8 +777,8 @@ func TestBuildVersion(t *testing.T) { func TestRunVersionWriteFailureIsFatal(t *testing.T) { t.Parallel() - // README §Error handling: a stdout write failure exits 1, reported - // in one line on stderr. + // README §Error handling: a failed write to stdout is a fatal error, + // exit 1, reported on stderr starting with "sfdupes:". var stderr bytes.Buffer code := run([]string{"--version"}, failingWriter{}, &stderr) @@ -796,7 +800,7 @@ func TestRunVersionWriteFailureIsFatal(t *testing.T) { func scanFixture(t *testing.T) []string { t.Helper() - dir := t.TempDir() + dir := realTempDir(t) dupes := []string{ writeFile(t, dir, "one/a.bin", pattern(1, 300)), @@ -817,6 +821,23 @@ func scanFixture(t *testing.T) []string { return dupes } +// storedDiskSize returns the disk_size of the record for path in the +// database at dbPath. A test of a trees run compares with this rather +// than with a new lstat, which on ZFS can differ from the scan's (see +// diskSizeOf). +func storedDiskSize(t *testing.T, dbPath, path string) int64 { + t.Helper() + + db, err := openReportDatabase(t.Context(), dbPath) + if err != nil { + t.Fatal(err) + } + + defer func() { _ = db.Close() }() + + return recordByPath(t, dbRecords(t, db), path).diskSize +} + // scanOK runs scan over operands, fails the test unless it exits 0 with // nothing on stdout, and returns everything it printed to stderr. func scanOK(t *testing.T, operands ...string) string { @@ -849,7 +870,7 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) { path := testDBPath(t) t.Setenv(databaseEnv, path) - dir := t.TempDir() + dir := realTempDir(t) writeFile(t, dir, "a.bin", pattern(1, 300)) // Same size as a.bin, so the scan reads it, and the read fails. @@ -877,75 +898,292 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) { assertNoSidecars(t, path) } -func TestRunScanSkipsSymlinkOperand(t *testing.T) { - path := testDBPath(t) - t.Setenv(databaseEnv, path) +// realTempDir returns a new temporary directory by its real path, so +// tests hold where the temporary directory is reached through a +// symlink. +func realTempDir(t *testing.T) string { + t.Helper() - dir := t.TempDir() - writeFile(t, dir, "target/sub/f", pattern(1, 10)) - - link := filepath.Join(dir, "link") - - err := os.Symlink(filepath.Join(dir, "target"), link) + dir, err := filepath.EvalSymlinks(t.TempDir()) if err != nil { t.Fatal(err) } - // Scanning a directory through the symlink stores a record beneath - // the symlink's own path for a file beneath its target. - scanOK(t, filepath.Join(link, "sub")) - - assertOperandSkipped(t, path, link, "symlink", - filepath.Join(link, "sub", "f")) + return dir } +// symlink creates link pointing at target. +func symlink(t *testing.T, target, link string) { + t.Helper() + + err := os.Symlink(target, link) + if err != nil { + t.Fatal(err) + } +} + +// TestRunScanFollowsSymlinkOperand scans link, a symlink to target. +// The operand is scanned at its real path, and its file is recorded +// there. +func TestRunScanFollowsSymlinkOperand(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := realTempDir(t) + f := writeFile(t, dir, "target/sub/f", pattern(1, 10)) + + link := filepath.Join(dir, "link") + symlink(t, filepath.Join(dir, "target"), link) + + stderr := scanOK(t, link) + + summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " + + "0 unchanged), 0 skipped\n" + if !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q", stderr, summary) + } + + if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { + t.Errorf("records = %q, want %q", got, []string{f}) + } +} + +// TestRunScanWalksOperandUnderSymlinkOperand scans link and link/sub, +// with link a symlink to target. Both resolve under target, so +// target/sub is walked once, as part of target. func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) { path := testDBPath(t) t.Setenv(databaseEnv, path) - dir := t.TempDir() - writeFile(t, dir, "target/sub/f", pattern(1, 10)) + dir := realTempDir(t) + f := writeFile(t, dir, "target/sub/f", pattern(1, 10)) link := filepath.Join(dir, "link") + symlink(t, filepath.Join(dir, "target"), link) - err := os.Symlink(filepath.Join(dir, "target"), link) - if err != nil { - t.Fatal(err) + stderr := scanOK(t, link, filepath.Join(link, "sub")) + + summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " + + "0 unchanged), 0 skipped\n" + if !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q", stderr, summary) } - // link is dropped as a symlink, but link/sub must still be scanned, - // not dropped as lying under link. - scanOK(t, link, filepath.Join(link, "sub")) - - db, err := openDB(path, reportParams) - if err != nil { - t.Fatal(err) + if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { + t.Errorf("records = %q, want %q", got, []string{f}) } - - t.Cleanup(func() { _ = db.Close() }) - - recordByPath(t, dbRecords(t, db), filepath.Join(link, "sub", "f")) } -func TestRunScanSkipsZFSOperand(t *testing.T) { +// TestRunScanWalksSymlinkedOperandBelowOperand is the case of +// https://git.eeqj.de/sneak/sfdupes/issues/144: scan dir dir/link/sub, +// with dir/link a symlink to a directory outside dir. The walk of dir +// does not follow link, so dir/link/sub is walked on its own at its +// real path, and the record an earlier scan stored there survives. +func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) { path := testDBPath(t) t.Setenv(databaseEnv, path) - zfs := filepath.Join(t.TempDir(), ".zfs") - snapshot := filepath.Join(zfs, "snapshot", "hourly") - f := writeFile(t, snapshot, "f", pattern(1, 10)) + dir := realTempDir(t) + a := writeFile(t, dir, "a", pattern(1, 10)) - // An operand beneath a .zfs directory is walked, because it is not - // itself named .zfs. - scanOK(t, snapshot) + outside := realTempDir(t) + f := writeFile(t, outside, "sub/f", pattern(2, 10)) + + symlink(t, outside, filepath.Join(dir, "link")) + + scanOK(t, filepath.Join(outside, "sub")) + + stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub")) + + summary := "scan: 2 files seen (1 added, 0 updated, 0 removed, " + + "1 unchanged), 0 skipped\n" + if !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q", stderr, summary) + } + + want := []string{a, f} + slices.Sort(want) + + if got := storedPaths(t, path); !slices.Equal(got, want) { + t.Errorf("records = %q, want %q", got, want) + } +} + +// TestRunScanDropsSymlinkOperandInsideOperand scans dir and link, with +// link, outside dir, a symlink to dir/real. link resolves inside dir, +// so it is dropped as covered by dir, and its file is recorded once. +func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := realTempDir(t) + f := writeFile(t, dir, "real/f", pattern(1, 10)) + + link := filepath.Join(realTempDir(t), "link") + symlink(t, filepath.Join(dir, "real"), link) + + stderr := scanOK(t, dir, link) + + summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " + + "0 unchanged), 0 skipped\n" + if !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q", stderr, summary) + } + + if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { + t.Errorf("records = %q, want %q", got, []string{f}) + } +} + +// TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and +// a symlink whose target does not exist. The scan fails with exit 1 and +// an error naming the symlink as given, before it walks anything, so no +// database is created. +func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := realTempDir(t) + writeFile(t, dir, "f", pattern(1, 10)) + + dangling := filepath.Join(realTempDir(t), "dangling") + symlink(t, filepath.Join(dir, "missing"), dangling) + + stdout := captureStdout(t) + + var stderr bytes.Buffer + + code := run([]string{cmdScan, dir, dangling}, os.Stdout, &stderr) + if code != exitFatal { + t.Errorf("run(scan %s %s) = %d, want %d", dir, dangling, code, + exitFatal) + } + + assertFatalOutput(t, stderr.String(), stdout()) + + want := "sfdupes: resolve " + dangling + ": " + if !strings.Contains(stderr.String(), want) { + t.Errorf("stderr = %q, want %q", stderr.String(), want) + } + + _, err := os.Stat(path) + if !errors.Is(err, fs.ErrNotExist) { + t.Errorf("database: stat error %v, want it missing", err) + } +} + +// TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath runs scan . +// after a cd through home/proj, a symlink to data/proj. The file is +// recorded under data/proj, its real path. +func TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := realTempDir(t) + f := writeFile(t, dir, "data/proj/f", pattern(1, 10)) + + err := os.Mkdir(filepath.Join(dir, "home"), 0o750) + if err != nil { + t.Fatal(err) + } + + symlink(t, filepath.Join(dir, "data", "proj"), + filepath.Join(dir, "home", "proj")) + + // t.Chdir also sets PWD to the path through the symlink, as a + // shell does after cd. + t.Chdir(filepath.Join(dir, "home", "proj")) + + scanOK(t, ".") + + if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { + t.Errorf("records = %q, want %q", got, []string{f}) + } +} + +// TestRunScanFollowsSymlinkBeforeDotDot scans a path with a ".." after +// home/proj, a symlink to data/proj: home/proj/.. from dir, and .. after +// a cd through home/proj. As with realpath, the ".." counts from +// data/proj, so data is scanned and home is not. +func TestRunScanFollowsSymlinkBeforeDotDot(t *testing.T) { + for _, tc := range []struct{ wd, operand string }{ + {"", "home/proj/.."}, + {"home/proj", ".."}, + } { + t.Run(tc.operand, func(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := realTempDir(t) + f := writeFile(t, dir, "data/proj/f", pattern(1, 10)) + g := writeFile(t, dir, "data/g", pattern(2, 10)) + writeFile(t, dir, "home/h", pattern(3, 10)) + symlink(t, filepath.Join(dir, "data", "proj"), + filepath.Join(dir, "home", "proj")) + + // t.Chdir also sets PWD to the path given, as a shell does + // after cd. + t.Chdir(filepath.Join(dir, tc.wd)) + + scanOK(t, tc.operand) + + want := []string{f, g} + slices.Sort(want) + + if got := storedPaths(t, path); !slices.Equal(got, want) { + t.Errorf("records = %q, want %q", got, want) + } + }) + } +} + +// TestRunScanSkipsSymlinkToFIFOOperand scans link, a symlink to a FIFO. +// The scan warns about the FIFO by its real path, counts it as skipped, +// and exits 0. +func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) { + t.Setenv(databaseEnv, testDBPath(t)) + + dir := realTempDir(t) + fifo := filepath.Join(dir, "fifo") + + err := unix.Mkfifo(fifo, 0o600) + if err != nil { + t.Fatal(err) + } + + link := filepath.Join(dir, "link") + symlink(t, fifo, link) + + stderr := scanOK(t, link) + + warning := "walk " + fifo + ": skipping FIFO operand\n" + if !strings.Contains(stderr, warning) { + t.Errorf("stderr = %q, want %q", stderr, warning) + } + + summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " + + "0 unchanged), 1 skipped\n" + if !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q", stderr, summary) + } +} + +func TestRunScanSkipsZFSOperand(t *testing.T) { + zfs := filepath.Join(realTempDir(t), ".zfs") + f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10)) + + // No scan records a file inside a .zfs directory, so the record + // beneath the operand is written straight into the database. + path := seedDatabase(t, []scanRec{{size: 10, path: f}}) + t.Setenv(databaseEnv, path) assertOperandSkipped(t, path, zfs, ".zfs directory", f) } // assertOperandSkipped scans operand alone and checks that it is skipped // as kind: a warning naming it, one skip in the summary, exit 0, and the -// record for kept, which an earlier scan stored beneath operand, still -// in the database at dbPath. +// record for kept, stored beneath operand before this scan, still in the +// database at dbPath. func assertOperandSkipped(t *testing.T, dbPath, operand, kind, kept string, ) { @@ -974,6 +1212,107 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind, recordByPath(t, dbRecords(t, db), kept) } +// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs +// directory, first alone and then under another operand. Each time it +// is warned about and counted as skipped, and nothing in it is walked. +// The other operand is scanned as usual. +func TestRunScanSkipsOperandInsideZFS(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := realTempDir(t) + hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly") + writeFile(t, hourly, "f", pattern(1, 10)) + f := writeFile(t, dir, "f", pattern(2, 10)) + + warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n" + + for _, tc := range []struct { + operands []string + summary string + }{ + { + []string{hourly}, + "scan: 0 files seen (0 added, 0 updated, 0 removed, " + + "0 unchanged), 1 skipped\n", + }, + { + []string{dir, hourly}, + "scan: 1 files seen (1 added, 0 updated, 0 removed, " + + "0 unchanged), 1 skipped\n", + }, + } { + stderr := scanOK(t, tc.operands...) + if !strings.Contains(stderr, warning) || + !strings.Contains(stderr, tc.summary) { + t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands, + stderr, warning, tc.summary) + } + } + + db, err := openDB(path, reportParams) + if err != nil { + t.Fatal(err) + } + + t.Cleanup(func() { _ = db.Close() }) + + if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) { + t.Errorf("records = %q, want %q", got, []string{f}) + } +} + +// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where +// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about +// by its real path, counted as skipped, and not walked. +func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) { + t.Setenv(databaseEnv, testDBPath(t)) + + dir := realTempDir(t) + writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10)) + + snaps := filepath.Join(dir, "snaps") + symlink(t, filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps) + + stderr := scanOK(t, filepath.Join(snaps, "daily")) + + daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily") + warning := "walk " + daily + ": skipping operand inside a .zfs directory\n" + summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " + + "0 unchanged), 1 skipped\n" + + if !strings.Contains(stderr, warning) || + !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary) + } +} + +// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is +// a symlink to pool/data. The operand is scanned as usual, and its file +// is recorded by its real path. +func TestRunScanWalksOperandThroughSymlink(t *testing.T) { + path := testDBPath(t) + t.Setenv(databaseEnv, path) + + dir := realTempDir(t) + f := writeFile(t, dir, "pool/data/daily/f", pattern(1, 10)) + + data := filepath.Join(dir, "data") + symlink(t, filepath.Join(dir, "pool", "data"), data) + + stderr := scanOK(t, filepath.Join(data, "daily")) + + summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " + + "0 unchanged), 0 skipped\n" + if !strings.Contains(stderr, summary) { + t.Errorf("stderr = %q, want %q", stderr, summary) + } + + if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { + t.Errorf("records = %q, want %q", got, []string{f}) + } +} + func TestRunReportSucceeds(t *testing.T) { path := testDBPath(t) t.Setenv(databaseEnv, path) @@ -1028,8 +1367,9 @@ func TestRunTreesSucceeds(t *testing.T) { // The two directories holding the duplicate pair are duplicate // trees of each other. - want := "first\tdupe\tfiles\tsize\n" + - filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n" + want := "first\tdupe\tfiles\tsize\tdisk_size\n" + + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" + + strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n" if got := stdout.String(); got != want { t.Errorf("stdout = %q, want %q", got, want) } @@ -1063,9 +1403,10 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) { cases := map[string]string{ cmdReport: "first\tdupe\tsize\n" + dupes[0] + "\t" + dupes[1] + "\t300\n", - cmdTrees: "first\tdupe\tfiles\tsize\n" + + cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + - "\t1\t300\n", + "\t1\t300\t" + + strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n", } for name, want := range cases { @@ -1100,8 +1441,11 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) { t.Errorf("report stdout = %q, want %q", got, want) } - want = "first\tdupe\tfiles\tsize\n" + - filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n" + diskSize := storedDiskSize(t, filepath.Join(dir, name), dupes[1]) + + want = "first\tdupe\tfiles\tsize\tdisk_size\n" + + filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" + + strconv.FormatInt(diskSize, 10) + "\n" if got := runStdout(t, cmdTrees); got != want { t.Errorf("trees stdout = %q, want %q", got, want) } @@ -1122,7 +1466,7 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) { } func TestRunDatabasePathUsedAsGiven(t *testing.T) { - // README §Database: the path names the database file exactly. In + // README §Settings: the path names the database file exactly. In // SQLite's connection string an unescaped ? or # would end the file // name and % would start an escape, and a path starting with // // could be read as a host name. %25 is a valid escape, so unescaped @@ -1167,9 +1511,8 @@ func holdScanLock(t *testing.T, path string) { } func TestRunSecondScanFails(t *testing.T) { - // README §Database: while one scan holds the lock, a second scan - // fails at once, naming the lock file, without opening the - // database. + // README §Database: a second scan that finds the lock held fails + // at once, naming the lock file, without opening the database. path := testDBPath(t) t.Setenv(databaseEnv, path) @@ -1228,7 +1571,7 @@ func assertSecondScanFails(t *testing.T, path string) { } func TestRunSecondScanThroughSymlinkFails(t *testing.T) { - // README §Database: every path to one database names the same lock + // README §Database: a path through a symlink names the same lock // file, so while a scan holds it through one path, a scan through // another fails, naming the lock file beside the database's real // path. @@ -1333,8 +1676,8 @@ func TestRunReportsDuringScan(t *testing.T) { } func TestRunStdoutClosedIsFatal(t *testing.T) { - // README §Error handling: a stdout write failure exits 1, reported - // in one line on stderr. + // README §Error handling: a failed write to stdout is a fatal error, + // exit 1, reported on stderr starting with "sfdupes:". for _, name := range []string{cmdReport, cmdTrees, cmdInfo} { t.Run(name, func(t *testing.T) { t.Setenv(databaseEnv, testDBPath(t)) diff --git a/progress.go b/progress.go index 6802825..3b112c3 100644 --- a/progress.go +++ b/progress.go @@ -9,6 +9,7 @@ import ( "strings" "sync" "time" + "unicode/utf8" "github.com/schollz/progressbar/v3" "golang.org/x/term" @@ -34,10 +35,17 @@ const stallTime = 10 * time.Second // spinnerType selects the progressbar library's spinner style. const spinnerType = 14 -// barWidth is the width of the progressbar library's bar between its -// two ends, as in the example line in README "Progress". +// barWidth is the width of a phase's bar between its two ends, as in +// the example line in README "Progress". const barWidth = 10 +// barCell is one column of the filled part of a bar. +const barCell = "█" + +// barColumns is the width of what percentBar returns: the percent in +// four columns and its sign, a space, and the bar with its two ends. +const barColumns = barWidth + 8 + // percentScale converts a fraction to a percentage. const percentScale = 100 @@ -45,19 +53,14 @@ const percentScale = 100 // read, as the progressbar library takes it. const fallbackWidth = 80 -// spinnerMargin is the columns of a spinner's line besides its text: -// the spinner and the three spaces the library draws around the text, -// and the terminal's last column, which every line stays clear of so -// that no terminal wraps it. +// spinnerMargin is the columns of a line besides its text: the spinner +// and the three spaces the library draws around the text, and the +// terminal's last column, which every line stays clear of so that no +// terminal wraps it. const spinnerMargin = 5 -// barMargin is the columns of a bar's line besides its text: the -// percent and the bar the library draws after the text, the spaces -// around them, and the terminal's last column. -const barMargin = barWidth + 11 - // countingRecords is what the display of a phase that reads the records -// says while it counts them, before it has its total. +// says until it has the number of records, its total. const countingRecords = "counting the records" // openPhase is the name of the phase in which report, trees and info @@ -85,7 +88,7 @@ func terminalWidth() int { return width } -// phaseLabel names phase with its place among phases, as "[2/5] walk". +// phaseLabel names phase with its place among phases, as "[2/4] walk". func phaseLabel(phases []string, phase string) string { return fmt.Sprintf("[%d/%d] %s", slices.Index(phases, phase)+1, len(phases), phase) @@ -109,13 +112,13 @@ type drawnCount struct { } // progress renders one phase's progress on stderr. On a TTY the -// progressbar library draws line, fitted to the terminal's width: after -// the library's spinner, or, for a phase with an exact total, before the -// library's bar when the terminal has room for the bar beside it. It is -// drawn again every redrawInterval, so its elapsed time keeps moving -// while the phase waits for its next item. When stderr is not a TTY it -// emits no ANSI redraws: it prints line as the phase starts, every -// plainInterval while it runs, and as it ends. +// progressbar library draws its spinner and then line, fitted to the +// terminal's width. A phase with an exact total ends the line with its +// percent and bar, at the terminal's right edge, when the terminal has +// room for them. It is drawn again every redrawInterval, so its elapsed +// time keeps moving while the phase waits for its next item. When +// stderr is not a TTY it emits no ANSI redraws: it prints line as the +// phase starts, every plainInterval while it runs, and as it ends. // // The draws come from a goroutine of their own, so mu guards the fields // below it. Every method must be called from the main goroutine. A nil @@ -131,7 +134,6 @@ type progress struct { mu sync.Mutex bar *progressbar.ProgressBar // what the library draws; nil until then - withBar bool // bar is the library's bar, not its spinner widest int // the widest line bar has drawn, in columns label string unit string @@ -141,7 +143,7 @@ type progress struct { stalledOn string count int64 start time.Time // when the phase started, for the elapsed time - rateStart time.Time // when the count started, for the ETA + rateStart time.Time // when the count started, for the time remaining lastItem time.Time // when the last item was completed, or the count started // drawn holds the count at each draw over the last rateWindow, oldest // first; the rate counts from the first. It keeps the draw before the @@ -215,55 +217,34 @@ func (p *progress) draw() { } if p.width == nil { - fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true)) + fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, false)) return } width := p.width() + text := p.line(now, width-spinnerMargin, true) - // A phase with an exact total gets the library's bar when its whole - // line, without the percent the bar shows, fits beside the bar: the - // bar gives way before any part of the line. A total of 0 leaves the - // bar nothing to fill. - text, withBar := "", false - if p.total > 0 && !p.estimate { - text = p.line(now, math.MaxInt, false) - withBar = len(text) <= width-barMargin - } - - margin := barMargin - if !withBar { - margin = spinnerMargin - text = p.line(now, width-margin, true) - } - - // The library makes a bar or a spinner, and before each draw erases - // its line with as many spaces as the widest line it has drawn. A - // display gets a new one when it turns from one to the other, as its - // line grows or the terminal is resized, and when the terminal has - // been made too narrow for that erase, which would wrap and leave a - // line behind at every draw. - if p.bar == nil || p.withBar != withBar || p.widest >= width { - p.newBar(withBar, width) - } - - if withBar { - _ = p.bar.Set64(p.count) + // Before each draw the library erases its line with as many spaces as + // the widest line it has drawn. A display gets a new one when the + // terminal has been made too narrow for that erase, which would wrap + // and leave a line behind at every draw. + if p.bar == nil || p.widest >= width { + p.newBar(width) } p.bar.Describe(text) - // The library drew the text and its margin, less the terminal's last - // column, which the margin keeps clear. - p.widest = max(p.widest, len(text)+margin-1) + // The library drew the spinner, the text and the spaces around it, + // less the terminal's last column, which spinnerMargin keeps clear. A + // bar's cells take one column each but more than one byte. + p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1) } -// newBar has the library make what draw draws: a bar for the phase's -// total when withBar is set, and a spinner when not. Either shows the -// text draw gives it, and nothing of its own besides the spinner, or -// the bar and its percent. p.mu must be held. -func (p *progress) newBar(withBar bool, width int) { +// newBar has the library make what draw draws: its spinner, followed by +// the text draw gives it and nothing of the library's own. p.mu must be +// held. +func (p *progress) newBar(width int) { if p.bar != nil { // Erase what was drawn before, short of the last column of a // terminal that may have been made narrower since: the library's @@ -272,29 +253,36 @@ func (p *progress) newBar(withBar bool, width int) { fmt.Fprint(os.Stderr, "\r"+blank+"\r") } - total := int64(-1) // the library's spinner - if withBar { - total = p.total - } - - p.bar = progressbar.NewOptions64(total, + // A total of -1 gives the library's spinner. + p.bar = progressbar.NewOptions64(-1, progressbar.OptionSetWriter(os.Stderr), - progressbar.OptionSetWidth(barWidth), progressbar.OptionSpinnerType(spinnerType), // The spinner turns at each draw, not on a timer of the // library's own. progressbar.OptionSetSpinnerChangeInterval(0), - // The text carries the elapsed time and the ETA. + // The text carries the elapsed time and the time remaining. progressbar.OptionSetElapsedTime(false), progressbar.OptionSetPredictTime(false), ) - p.withBar, p.widest = withBar, 0 + p.widest = 0 +} + +// percentBar returns the percent and the bar that end the text of a +// phase with an exact total, as " 25% |██ |": the bar's +// barWidth columns fill in proportion to the items done. A count past +// the total, as when a scan adds records while trees reads them, shows +// as 100%. p.mu must be held. +func (p *progress) percentBar() string { + done := min(float64(p.count)/float64(p.total), 1) + filled := strings.Repeat(barCell, int(done*barWidth)) + + return fmt.Sprintf("%4.0f%% |%-*s|", done*percentScale, barWidth, filled) } // setTotal gives a phase that started without a total its total, once -// it is known. The rate, the ETA and the time with no item completed -// count from then on: the time spent waiting for the total says nothing -// about how fast items go. +// it is known. The rate, the time remaining and the time with no item +// completed count from then on: the time spent waiting for the total +// says nothing about how fast items go. func (p *progress) setTotal(total int64) { if p == nil { return @@ -438,22 +426,26 @@ func (p *progress) finish() { // line formats the phase's progress as of now in at most room columns, // as // -// [3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s +// [3/4] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, remain 13s // -// where "files" is the phase's unit, without the percent when percent -// is false, as before the library's bar, which shows its own. A phase -// with no total shows the items done instead, or what it waits for. The -// rate is that since the first draw in drawn, and the ETA comes from -// the rate since the count started. Once no item has been completed for -// stallTime, the rate, which would read 0, gives way to the word -// "stalled" and where the phase waits, as "stalled disk/db". +// where "files" is the phase's unit. A phase with no total shows the +// items done instead, or what it waits for. The rate is that since the +// first draw in drawn, and the time remaining comes from the rate since +// the count started. Once no item has been completed for stallTime, the +// rate, which would read 0, gives way to the word "stalled" and where +// the phase waits, as "stalled disk/db". // -// The phase and its place, the progress, the elapsed time and the ETA -// are kept whole. A line wider than room leaves out the rate, the words -// of a stalled phase, or what the phase waits for; only if it is still -// wider is it cut at room. Every part is ASCII, so its length is its -// width in columns. p.mu must be held. -func (p *progress) line(now time.Time, room int, percent bool) string { +// When bar is true, as on a terminal, a phase with an exact total ends +// the line with its percent and bar, in place of the percent after its +// count. A total of 0 leaves the bar nothing to fill. +// +// The phase and its place, the progress, the elapsed time and the time +// remaining are kept whole. A line wider than room first leaves out the +// rate, the words of a stalled phase, or what the phase waits for. Next +// it leaves out the bar. Only if it is still wider is it cut at room. +// Every part but the bar is ASCII, so its length is its width in +// columns. p.mu must be held. +func (p *progress) line(now time.Time, room int, bar bool) string { elapsed := now.Sub(p.start).Round(time.Second) // head and tail hold the parts kept whole; extra, between them, is @@ -462,21 +454,41 @@ func (p *progress) line(now time.Time, room int, percent bool) string { if p.total < 0 && p.waitingFor != "" { head, extra = p.label+": ", p.waitingFor+", " - tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed) + tail = fmt.Sprintf("elapsed %s, remain unknown", elapsed) } else { since := p.drawn[0] rate := perSecond(p.count-since.count, now.Sub(since.at)) average := perSecond(p.count, now.Sub(p.rateStart)) - head = p.label + ": " + p.itemsDone(percent) + ", " + head = p.label + ": " + p.itemsDone(true) + ", " extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit) - tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average)) + tail = fmt.Sprintf("elapsed %s, remain %s", elapsed, p.remain(average)) if now.Sub(p.lastItem) >= stallTime { extra = "stalled " + p.stalledOn + ", " } } + if bar && p.total > 0 && !p.estimate { + // Beside the bar, which shows the percent, the count shows none. + // The line is padded to the room the bar leaves it, so the bar + // stays at room's right edge however long the line is. + beside := room - barColumns + barHead := p.label + ": " + p.itemsDone(false) + ", " + + switch { + case len(barHead+extra+tail) <= beside: + return fmt.Sprintf("%-*s%s", beside, barHead+extra+tail, + p.percentBar()) + case len(barHead+tail) <= beside: + return fmt.Sprintf("%-*s%s", beside, barHead+tail, p.percentBar()) + } + + // The bar gives way. The rate, or the words of a stalled phase, + // gave way before it and stay out. + extra = "" + } + full, kept := head+extra+tail, head+tail switch { @@ -526,10 +538,10 @@ func (p *progress) itemsDone(percent bool) string { return fmt.Sprintf("%s %.0f%%", done, pct) } -// eta returns the time the phase has left at rate items a second, or +// remain returns the time the phase has left at rate items a second, or // "unknown": with no total, before the first item, and once the count // passes an estimate. p.mu must be held. -func (p *progress) eta(rate float64) string { +func (p *progress) remain(rate float64) string { switch { case p.total < 0, p.estimate && p.count > p.total: return "unknown" @@ -539,8 +551,8 @@ func (p *progress) eta(rate float64) string { return "unknown" } - remaining := time.Duration( + left := time.Duration( float64(p.total-p.count) / rate * float64(time.Second)) - return remaining.Round(time.Second).String() + return left.Round(time.Second).String() } diff --git a/progress_test.go b/progress_test.go index 1c65053..4093aa6 100644 --- a/progress_test.go +++ b/progress_test.go @@ -23,7 +23,7 @@ const testRedraw = time.Millisecond const testDeadline = 5 * time.Second // hashStalledOn is where the hash phase says it waits once it has -// stalled, as readRuns gives it. +// stalled, as hashPhase gives it. const hashStalledOn = "disk/db" // testClock is a clock a test moves by hand, for a display that reads @@ -117,38 +117,38 @@ func TestProgressLine(t *testing.T) { label: scanPhase("load"), total: -1, waitingFor: countingRecords, }, - want: "[1/5] load: counting the records, elapsed 10s, eta unknown", + want: "[1/4] load: counting the records, elapsed 10s, remain unknown", }, { p: &progress{ label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20, }, - want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s", + want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100}, - want: "[3/5] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown", + want: "[3/4] hash: [0/100] 0%, 0 files/s, elapsed 10s, remain unknown", }, { p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0}, - want: "[4/5] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s", + want: "[4/4] update: [0/0] 100%, 0 files/s, elapsed 10s, remain 0s", }, { p: &progress{ label: scanPhase("walk"), unit: scanUnit, total: 100, estimate: true, count: 20, }, - want: "[2/5] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s", + want: "[2/4] walk: [20/~100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { // Past the estimate the walk cannot tell how long it has left. p: &progress{ label: scanPhase("walk"), unit: scanUnit, total: 100, estimate: true, count: 120, }, - want: "[2/5] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " + - "eta unknown", + want: "[2/4] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " + + "remain unknown", }, { // A first scan's walk has no estimate. p: &progress{ label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20, }, - want: "[2/5] walk: 20 files, 2 files/s, elapsed 10s, eta unknown", + want: "[2/4] walk: 20 files, 2 files/s, elapsed 10s, remain unknown", }, { // The rate is in what the phase counts. p: &progress{ @@ -156,14 +156,14 @@ func TestProgressLine(t *testing.T) { count: 20, }, want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " + - "eta 40s", + "remain 40s", }} for _, tc := range cases { tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now tc.p.drawn = []drawnCount{{at: ago}} - if got := tc.p.line(now, math.MaxInt, true); got != tc.want { + if got := tc.p.line(now, math.MaxInt, false); got != tc.want { t.Errorf("line = %q, want %q", got, tc.want) } } @@ -184,9 +184,9 @@ func TestLineFitsRoom(t *testing.T) { start: ago, rateStart: ago, lastItem: now, drawn: []drawnCount{{at: ago}}, } - content := &progress{ - label: scanPhase("content"), total: -1, - waitingFor: "searching the database", start: ago, rateStart: ago, + load := &progress{ + label: scanPhase("load"), total: -1, + waitingFor: countingRecords, start: ago, rateStart: ago, } started := now.Add(-50 * time.Second) @@ -198,48 +198,49 @@ func TestLineFitsRoom(t *testing.T) { } cases := []struct { - p *progress - room int - percent bool - want string + p *progress + room int + bar bool + want string }{{ - p: hash, room: 57, percent: true, - want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s", + p: hash, room: 60, + want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { - p: hash, room: 56, percent: true, - want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40s", + p: hash, room: 59, + want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40s", }, { - p: hash, room: 45, percent: true, - want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40", + p: hash, room: 48, + want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40", }, { - // Before the library's bar, which shows the percent. - p: hash, room: 57, percent: false, - want: "[3/5] hash: [20/100], 2 files/s, elapsed 10s, eta 40s", + // The bar shows the percent. + p: hash, room: 74, bar: true, + want: "[3/4] hash: [20/100], 2 files/s, elapsed 10s, remain 40s" + + " 20% |██ |", }, { - p: content, room: 40, percent: true, - want: "[5/5] content: elapsed 10s, eta unknown", + p: load, room: 43, + want: "[1/4] load: elapsed 10s, remain unknown", }, { - p: stalled, room: 65, percent: true, - want: "[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " + - "eta 3m20s", + p: stalled, room: 68, + want: "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " + + "remain 3m20s", }, { - p: stalled, room: 64, percent: true, - want: "[3/5] hash: [20/100] 20%, elapsed 50s, eta 3m20s", + p: stalled, room: 67, + want: "[3/4] hash: [20/100] 20%, elapsed 50s, remain 3m20s", }} for _, tc := range cases { - if got := tc.p.line(now, tc.room, tc.percent); got != tc.want { - t.Errorf("line(%d, %t) = %q, want %q", - tc.room, tc.percent, got, tc.want) + if got := tc.p.line(now, tc.room, tc.bar); got != tc.want { + t.Errorf("line(%d, %t) = %q, want %q", tc.room, tc.bar, got, + tc.want) } } } // TestSetTotalStartsRate checks that setTotal starts a phase's rate, -// its ETA and its time with no record read from when the total becomes -// known, not from when the phase started waiting for it: after 20 -// seconds of waiting for the total, the phase has not stalled 5 seconds -// after it, and has 10 seconds after it with no record read. +// its time remaining and its time with no record read from when the +// total becomes known, not from when the phase started waiting for it: +// after 20 seconds of waiting for the total, the phase has not stalled 5 +// seconds after it, and has 10 seconds after it with no record read. func TestSetTotalStartsRate(t *testing.T) { t.Parallel() @@ -255,15 +256,15 @@ func TestSetTotalStartsRate(t *testing.T) { clock.Add(5 * time.Second) - want := "[1/5] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown" - if got := p.line(clock.Now(), math.MaxInt, true); got != want { + want := "[1/4] load: [0/100] 0%, 0 files/s, elapsed 25s, remain unknown" + if got := p.line(clock.Now(), math.MaxInt, false); got != want { t.Errorf("line = %q, want %q", got, want) } clock.Add(5 * time.Second) - want = "[1/5] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown" - if got := p.line(clock.Now(), math.MaxInt, true); got != want { + want = "[1/4] load: [0/100] 0%, stalled db, elapsed 30s, remain unknown" + if got := p.line(clock.Now(), math.MaxInt, false); got != want { t.Errorf("line = %q, want %q", got, want) } @@ -272,8 +273,8 @@ func TestSetTotalStartsRate(t *testing.T) { p.increment() } - want = "[1/5] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s" - if got := p.line(clock.Now(), math.MaxInt, true); got != want { + want = "[1/4] load: [20/100] 20%, 2 files/s, elapsed 30s, remain 40s" + if got := p.line(clock.Now(), math.MaxInt, false); got != want { t.Errorf("line = %q, want %q", got, want) } } @@ -316,9 +317,9 @@ func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) { lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n") - want := "[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown" + want := "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown" if len(lines) != 2 || lines[0] != want || - !strings.HasPrefix(lines[1], "[3/5] hash: [1000/1000] 100%, ") { + !strings.HasPrefix(lines[1], "[3/4] hash: [1000/1000] 100%, ") { t.Errorf("stderr lines = %q, want %q, then the line at 1000", lines, want) } @@ -336,7 +337,7 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) { }, nil, testRedraw, time.Now) awaitOutput(t, stderr, - "[1/5] load: counting the records, elapsed 0s, eta unknown\n", 3) + "[1/4] load: counting the records, elapsed 0s, remain unknown\n", 3) p.finish() } @@ -366,23 +367,23 @@ func TestPlainProgressStalledProducer(t *testing.T) { clock.Add(5 * time.Second) awaitOutput(t, stderr, - "[3/5] hash: [20/100] 20%, 4 files/s, elapsed 5s, eta 20s\n", 3) + "[3/4] hash: [20/100] 20%, 4 files/s, elapsed 5s, remain 20s\n", 3) clock.Add(time.Second) awaitOutput(t, stderr, - "[3/5] hash: [20/100] 20%, 0 files/s, elapsed 6s, eta 24s\n", 3) + "[3/4] hash: [20/100] 20%, 0 files/s, elapsed 6s, remain 24s\n", 3) clock.Add(4 * time.Second) awaitOutput(t, stderr, - "[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n", + "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 10s, remain 40s\n", 3) } // TestPlainProgressRateAfterStall draws a phase's plain line by hand, a // little over plainInterval after the line before, as a late tick // does, through a stall and a burst of files after it: the rate covers -// the files read since the line before, while the ETA comes from the -// phase's average. +// the files read since the line before, while the time remaining comes +// from the phase's average. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestPlainProgressRateAfterStall(t *testing.T) { @@ -410,10 +411,10 @@ func TestPlainProgressRateAfterStall(t *testing.T) { p.finish() want := []string{ - "[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown", - "[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown", - "[3/5] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown", - "[3/5] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s", + "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown", + "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 5s, remain unknown", + "[3/4] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, remain unknown", + "[3/4] hash: [100/1000] 10%, 20 files/s, elapsed 15s, remain 2m15s", } lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n") @@ -426,8 +427,8 @@ func TestPlainProgressRateAfterStall(t *testing.T) { // redrawInterval, as a terminal does, through a stall of stallTime and // a burst of files after it: the rate covers the files read over the // last rateWindow, and falls to 0 once the burst is older than that, -// while the ETA comes from the phase's average. Plain lines show each -// draw. +// while the time remaining comes from the phase's average. Plain lines +// show each draw. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressRateAfterStallRedrawn(t *testing.T) { @@ -457,8 +458,8 @@ func TestProgressRateAfterStallRedrawn(t *testing.T) { redraw(rateWindow + redrawInterval) p.finish() - want := "[3/5] hash: [50/1000] 5%, 10 files/s, elapsed 15s, eta 4m45s\n" + - "[3/5] hash: [50/1000] 5%, 0 files/s, elapsed 15s, eta 4m47s\n" + want := "[3/4] hash: [50/1000] 5%, 10 files/s, elapsed 15s, remain 4m45s\n" + + "[3/4] hash: [50/1000] 5%, 0 files/s, elapsed 15s, remain 4m47s\n" if !strings.Contains(stderr(), want) { t.Errorf("stderr = %q, want the lines %q", stderr(), want) } @@ -480,10 +481,10 @@ func lastFrame(out string) string { } // TestTerminalRedrawsWhileWaiting checks that the terminal display of a -// phase, spinner or bar, keeps being drawn while the phase waits for its -// next item, with the elapsed time as of now, and shows every item -// completed before the wait. stderr is not a terminal here, so the -// displays take it to be fallbackWidth wide. +// phase, with a bar or without, keeps being drawn while the phase waits +// for its next item, with the elapsed time as of now, and shows every +// item completed before the wait. The terminal is 120 columns wide, so +// the hash phase's display has room for its bar. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalRedrawsWhileWaiting(t *testing.T) { @@ -506,7 +507,8 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) { for _, tc := range cases { before := len(stderr()) since := func() string { return stderr()[before:] } - p := startProgress(tc.params, terminalWidth, testRedraw, time.Now) + p := startProgress(tc.params, func() int { return 120 }, testRedraw, + time.Now) for range 50 { p.increment() @@ -552,13 +554,13 @@ func drawLate(stderr func() string, params progressParams, width int, return stderr()[before:] } -// TestTerminalLinesKeepETA checks the line each display shows on an -// 80-column terminal at a large scan's counts and times, and the line -// of trees' read phase at the same counts: the rate is left out, so -// that the elapsed time and the ETA stay whole. +// TestTerminalLinesKeepTimeRemaining checks the line each display shows +// on an 80-column terminal at a large scan's counts and times, and the +// line of trees' read phase at the same counts: the rate is left out, +// so that the elapsed time and the time remaining stay whole. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr -func TestTerminalLinesKeepETA(t *testing.T) { +func TestTerminalLinesKeepTimeRemaining(t *testing.T) { stderr := captureStderr(t) cases := []struct { @@ -570,26 +572,20 @@ func TestTerminalLinesKeepETA(t *testing.T) { label: scanPhase("walk"), unit: scanUnit, total: 1200000, estimate: true, }, - want: "[2/5] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " + - "eta unknown", + want: "[2/4] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " + + "remain unknown", }, { params: progressParams{ label: scanPhase("hash"), unit: scanUnit, total: 9876543, }, - want: "[3/5] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " + - "eta 17h30m0s", - }, { - params: progressParams{ - label: scanPhase("content"), unit: scanUnit, total: 9876543, - }, - want: "[5/5] content: [1234567/9876543] 12%, elapsed 2h30m0s, " + - "eta 17h30m0s", + want: "[3/4] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " + + "remain 17h30m0s", }, { params: progressParams{ label: treesPhase("read"), unit: "records", total: 9876543, }, want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " + - "eta 17h30m0s", + "remain 17h30m0s", }} for _, tc := range cases { @@ -600,30 +596,32 @@ func TestTerminalLinesKeepETA(t *testing.T) { } } -// TestTerminalStalledLines draws every phase's line on an 80-column -// terminal once the phase has stalled, its last item done 42 seconds -// before. At an ordinary scan's counts and times, 12345 of 45678 items -// done four and a half minutes in, the line shows where the phase waits -// whole, beside the phase, progress, elapsed time and ETA. At a large -// scan's, 1234567 of 9876543 two and a half hours in, those four still -// show whole, as the words give way first where they do not fit. +// TestTerminalStalledLines draws every phase's line once the phase has +// stalled, its last item done 42 seconds before. At an ordinary scan's +// counts and times, 12345 of 45678 items done four and a half minutes +// in, on a terminal 100 columns wide, the line shows where the phase +// waits whole, beside the phase, progress, elapsed time and time +// remaining. At a large scan's, 1234567 of 9876543 two and a half hours +// in, on an 80-column terminal, those four still show whole, as the +// words give way first where they do not fit. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalStalledLines(t *testing.T) { stderr := captureStderr(t) - // drawStalled draws the display params describes as it starts and - // once more as it ends, with count of total items done elapsed into - // the phase, and returns what the terminal shows. - drawStalled := func(params progressParams, total, count int64, - elapsed time.Duration, + // drawStalled draws the display params describes on a terminal width + // columns wide as it starts and once more as it ends, with count of + // total items done elapsed into the phase, and returns what the + // terminal shows. + drawStalled := func(params progressParams, width int, total, + count int64, elapsed time.Duration, ) string { before := len(stderr()) clock := &testClock{now: time.Now()} params.total = total // No timed draws come between the two. - p := startProgress(params, func() int { return 80 }, time.Hour, + p := startProgress(params, func() int { return width }, time.Hour, clock.Now) clock.Add(elapsed - 42*time.Second) @@ -643,9 +641,6 @@ func TestTerminalStalledLines(t *testing.T) { {label: scanPhase("walk"), estimate: true, stalledOn: "disk"}, {label: scanPhase("hash"), stalledOn: hashStalledOn}, {label: scanPhase("update"), stalledOn: "db"}, - // The content phase's checks and reads, then its last write. - {label: scanPhase("content"), stalledOn: "disk/db"}, - {label: scanPhase("content"), stalledOn: "db"}, {label: treesPhase("read"), stalledOn: "db"}, {label: treesPhase("group"), stalledOn: "cpu"}, {label: treesPhase("write"), stalledOn: "out"}, @@ -658,19 +653,22 @@ func TestTerminalStalledLines(t *testing.T) { approx = "~" } - shown := drawStalled(params, 45678, 12345, + // The percent is in the line, or beside the bar. + shown := drawStalled(params, 100, 45678, 12345, 4*time.Minute+30*time.Second) - want := params.label + ": [12345/" + approx + "45678] 27%, stalled " + - params.stalledOn + ", elapsed 4m30s, eta 12m9s" - - if !strings.Contains(shown, want) { - t.Errorf("terminal shows %q, want %q", shown, want) + for _, want := range []string{ + params.label + ": [12345/" + approx + "45678]", + ", stalled " + params.stalledOn + ", elapsed 4m30s, remain 12m9s", + } { + if !strings.Contains(shown, want) { + t.Errorf("terminal shows %q, want %q", shown, want) + } } - shown = drawStalled(params, 9876543, 1234567, 150*time.Minute) + shown = drawStalled(params, 80, 9876543, 1234567, 150*time.Minute) for _, want := range []string{ params.label + ": [1234567/" + approx + "9876543] 12%, ", - ", elapsed 2h30m0s, eta 17h30m0s", + ", elapsed 2h30m0s, remain 17h30m0s", } { if !strings.Contains(shown, want) { t.Errorf("terminal shows %q, want %q", shown, want) @@ -693,7 +691,7 @@ func TestTerminalLinesFitWidth(t *testing.T) { for _, width := range []int{120, 100, 80, 72, 60, 50, 40} { for _, total := range []int64{9876543, 20000} { params := progressParams{ - label: scanPhase("content"), unit: scanUnit, total: total, + label: scanPhase("update"), unit: scanUnit, total: total, } out := drawLate(stderr, params, width, total/8) @@ -715,13 +713,164 @@ func TestTerminalLinesFitWidth(t *testing.T) { } } +// TestTerminalLinesStartWithSpinner draws a display with no total and +// one with a bar, two and a half hours into the phase, on a terminal 120 +// columns wide. Each line must start with the library's spinner in the +// first column, then a space and the phase's label. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalLinesStartWithSpinner(t *testing.T) { + stderr := captureStderr(t) + + for _, params := range []progressParams{ + {label: scanPhase("walk"), unit: scanUnit, total: -1}, + {label: scanPhase("hash"), unit: scanUnit, total: 9876543}, + } { + shown := lastFrame(drawLate(stderr, params, 120, 1234567)) + spinner, rest, _ := strings.Cut(shown, " ") + + if utf8.RuneCountInString(spinner) != 1 || + !strings.HasPrefix(rest, params.label+": ") { + t.Errorf("terminal shows %q, want the spinner, a space and %q", + shown, params.label) + } + + if params.total > 0 && !strings.Contains(shown, "% |") { + t.Errorf("terminal shows %q, want a bar", shown) + } + } +} + +// TestTerminalBarStaysAtRightEdge draws the hash phase's display two +// and a half hours in, at two counts whose lines differ in length, on +// terminals 110 and 120 columns wide. The bar is partly filled at the +// first count and full at the second. It must start the same number of +// columns from the terminal's right edge each time, and end the same +// number of columns from it: a line that grows or shrinks, as its time +// remaining does, never moves the bar, and nor do the bar's filled +// cells. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalBarStaysAtRightEdge(t *testing.T) { + stderr := captureStderr(t) + params := progressParams{ + label: scanPhase("hash"), unit: scanUnit, total: 9876543, + } + + widths := []int{110, 120} + counts := []int64{1234567, 9876543} + startFromEdge := make([]int, 0, len(widths)*len(counts)) + endFromEdge := make([]int, 0, len(widths)*len(counts)) + + for _, width := range widths { + for _, count := range counts { + shown := lastFrame(drawLate(stderr, params, width, count)) + + beforeBar, _, found := strings.Cut(shown, "% |") + if !found { + t.Fatalf("terminal %d wide shows %q, want a bar", width, shown) + } + + // The bar ends with the last "|". + throughBar := shown[:strings.LastIndex(shown, "|")+1] + + startFromEdge = append(startFromEdge, + width-utf8.RuneCountInString(beforeBar)) + endFromEdge = append(endFromEdge, + width-utf8.RuneCountInString(throughBar)) + } + } + + if slices.Min(startFromEdge) != slices.Max(startFromEdge) { + t.Errorf("bar starts %v columns from the right edge, want the same "+ + "every time", startFromEdge) + } + + if slices.Min(endFromEdge) != slices.Max(endFromEdge) { + t.Errorf("bar ends %v columns from the right edge, want the same "+ + "every time", endFromEdge) + } +} + +// TestTerminalRateGivesWayBeforeBar draws the hash phase's display two +// and a half hours in. On a terminal 90 columns wide the line fits +// beside the bar without its rate, but not with it: the bar is drawn +// and the rate is not. On a terminal 85 columns wide the line does not +// fit beside the bar even without its rate: neither is drawn, though +// the rate would fit where the bar was. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalRateGivesWayBeforeBar(t *testing.T) { + stderr := captureStderr(t) + params := progressParams{ + label: scanPhase("hash"), unit: scanUnit, total: 9876543, + } + + shown := lastFrame(drawLate(stderr, params, 90, 1234567)) + if !strings.Contains(shown, "% |") || strings.Contains(shown, "files/s") { + t.Errorf("terminal 90 wide shows %q, want the bar and no rate", shown) + } + + shown = lastFrame(drawLate(stderr, params, 85, 1234567)) + if strings.Contains(shown, "% |") || strings.Contains(shown, "files/s") { + t.Errorf("terminal 85 wide shows %q, want neither bar nor rate", shown) + } +} + +// TestTerminalSpinnerTurnsBesideBar draws the hash phase's display, its +// bar partly filled, several times on a terminal 120 columns wide. The +// spinner must turn at every draw. A filled cell of the bar takes one +// column but three bytes. A line measured in bytes would seem too wide +// for the terminal, and every draw would start the display over, its +// spinner included. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalSpinnerTurnsBesideBar(t *testing.T) { + stderr := captureStderr(t) + params := progressParams{ + label: scanPhase("hash"), unit: scanUnit, total: 9876543, + } + + // No timed draws: the test draws. + p := startProgress(params, func() int { return 120 }, time.Hour, + time.Now) + + p.mu.Lock() + p.count = 1234567 + p.mu.Unlock() + + const draws = 4 + + spinners := make([]string, 0, draws) + + for range draws { + before := len(stderr()) + + p.draw() + + spinner, _, _ := strings.Cut(lastFrame(stderr()[before:]), " ") + spinners = append(spinners, spinner) + } + + p.finish() + + for i := 1; i < len(spinners); i++ { + if spinners[i] == spinners[i-1] { + t.Errorf("spinners at successive draws are %q, want each to "+ + "differ from the one before", spinners) + + break + } + } +} + // TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two // and a half hours into the phase, on a terminal 120 columns wide, then // makes the terminal narrower than that line and draws it again a few // times. Nothing written after that, the erase before each redraw // included, may reach the narrower terminal's last column: it would -// wrap, and every redraw would leave a line behind. At these widths the -// spinner stays a spinner and the bar gives way to the spinner. +// wrap, and every redraw would leave a line behind. At the narrower +// width the hash phase's bar gives way. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { @@ -739,7 +888,7 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { progressParams{ label: scanPhase("hash"), unit: scanUnit, total: 9876543, }, - 90, + 80, }, } @@ -779,10 +928,10 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { } } -// TestProgressWarningsOnOwnLines drives the terminal displays, spinner -// and bar, through a run of warnings with no items between them, as when -// the walk meets many unreadable paths, for many redraws: every warning -// must land on a line of its own, never inside a redraw. +// TestProgressWarningsOnOwnLines drives the terminal displays, with a +// bar and without, through a run of warnings with no items between +// them, as when the walk meets many unreadable paths, for many redraws: +// every warning must land on a line of its own, never inside a redraw. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressWarningsOnOwnLines(t *testing.T) { @@ -828,11 +977,12 @@ func TestProgressWarningsOnOwnLines(t *testing.T) { } } -// TestProgressWriterRowsOnOwnLines drives the terminal displays, -// spinner and bar, through a run of rows written through the display's +// TestProgressWriterRowsOnOwnLines drives the terminal displays, with a +// bar and without, through a run of rows written through the display's // writer to the same terminal, as stdout is in a plain "sfdupes trees", // each written out at once, for many redraws: the terminal must show -// every row whole, in order, on a line of its own. +// every row whole, in order, on a line of its own. The terminal is 120 +// columns wide, so the write phase's display has room for its bar. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressWriterRowsOnOwnLines(t *testing.T) { @@ -849,7 +999,8 @@ func TestProgressWriterRowsOnOwnLines(t *testing.T) { var want []string for _, params := range cases { - p := startProgress(params, terminalWidth, testRedraw, time.Now) + p := startProgress(params, func() int { return 120 }, testRedraw, + time.Now) out := p.writer(os.Stderr) // No pause between rows: one is garbled only if a redraw lands diff --git a/report.go b/report.go index a3396fe..0504cd8 100644 --- a/report.go +++ b/report.go @@ -2,6 +2,7 @@ package main import ( "context" + "crypto/sha256" "database/sql" "fmt" "io" @@ -20,14 +21,16 @@ const minGroupSize = 2 // scanRec is one file record from the database. The signature (size, // head, tail, content) is the duplicate key; mtime is informational -// only and used by scan for change detection. +// only and used by scan for change detection. diskSize is the bytes the +// file takes on disk, which trees adds up. type scanRec struct { - size int64 - mtime time.Time - head string - tail string - content string - path string + size int64 + diskSize int64 + mtime time.Time + head [sha256.Size]byte + tail [sha256.Size]byte + content [sha256.Size]byte + path string } // reportPhase names one phase of report with its place among all of @@ -44,11 +47,11 @@ func reportPhase(name string) string { // problem, including a missing database, is fatal. // // The work runs in two phases, each under a progress display of its -// own on stderr: open (open the database) and read (count the records, -// then read them in signature order, writing each size's duplicate -// groups to stdout once that size is read; see readReport). Each -// display ends before the next phase starts, and before an error is -// returned to be printed. +// own on stderr: open (open the database) and read (read the number of +// records from the totals table, then read the records in signature +// order, writing each size's duplicate groups to stdout once that size +// is read; see readReport). Each display ends before the next phase +// starts, and before an error is returned to be printed. func runReport(ctx context.Context, stdout io.Writer) error { dbPath, source, err := databasePath() if err != nil { @@ -169,12 +172,6 @@ type reportWriter struct { func (w *reportWriter) add(r scanRec) error { w.records++ - // A record without a content hash has unknown content (README - // "Database"), so it is never a duplicate. - if r.content == "" { - return nil - } - if r.size == w.sig.size && r.head == w.sig.head && r.tail == w.sig.tail && r.content == w.sig.content { w.paths = append(w.paths, r.path) diff --git a/report_test.go b/report_test.go index bb27652..2023cb3 100644 --- a/report_test.go +++ b/report_test.go @@ -2,6 +2,7 @@ package main import ( "bytes" + "crypto/sha256" "database/sql" "errors" "fmt" @@ -18,14 +19,27 @@ import ( // awkwardDir is a directory name holding every byte the reports escape. const awkwardDir = "/d/\tone\ntwo\rthree\\four" +// testHash returns a stand-in hash for a test record: the bytes of s, +// then zeros. Stand-ins sort as their strings do, so a test can choose +// the order in which report reads its records. +func testHash(s string) [sha256.Size]byte { + var h [sha256.Size]byte + + copy(h[:], s) + + return h +} + // awkwardPairRecs is a duplicate pair in sibling directories /d/A and // awkwardDir. A raw tab sorts before "A" but its escaped form `\t` // sorts after it, so awkwardDir coming first shows that sorting uses // the raw path. func awkwardPairRecs() []scanRec { + h, tl, c := testHash("h"), testHash("t"), testHash("c") + return []scanRec{ - {size: 5, head: "h", tail: "t", content: "c", path: "/d/A/f"}, - {size: 5, head: "h", tail: "t", content: "c", path: awkwardDir + "/f"}, + {size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: "/d/A/f"}, + {size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: awkwardDir + "/f"}, } } @@ -143,8 +157,8 @@ func TestReportStdoutFailsWhileReading(t *testing.T) { recs := make([]scanRec, 2*ioBufSize/len(dir)) for i := range recs { recs[i] = scanRec{ - size: int64(i/2) + 1, head: "h", tail: "t", content: "c", - path: fmt.Sprintf("%s/%d", dir, i), + size: int64(i/2) + 1, head: testHash("h"), tail: testHash("t"), + content: testHash("c"), path: fmt.Sprintf("%s/%d", dir, i), } } @@ -173,7 +187,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) { // reportOpenLine is the plain line of report's open phase when it opens // the database at once; it prints it as the phase starts and as it ends. const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" // smokeReport is report's stdout for the smoke-test records. const smokeReport = "first\tdupe\tsize\n" + @@ -211,7 +225,7 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) { "report: database " + path + " (from " + databaseEnv + ")", reportOpenLine, reportOpenLine, - "[2/2] read: counting the records, elapsed 0s, eta unknown", + "[2/2] read: counting the records, elapsed 0s, remain unknown", "[2/2] read: [6/6] 100%, ", "report: 6 records read, 2 duplicate groups, 4 dupe files, " + "6.1 KiB reclaimable", @@ -277,13 +291,14 @@ func TestReadReportOnTerminal(t *testing.T) { func TestRunReportsIgnoreInsertionOrder(t *testing.T) { // README §Constraints: identical database contents give identical // output, whatever order the records were inserted in. + a, b := testHash("a"), testHash("b") + recs := append(smokeTreeRecs(), awkwardPairRecs()...) recs = append(recs, - scanRec{size: 50, head: "b", tail: "b", content: "b", path: "/y/2"}, - scanRec{size: 50, head: "b", tail: "b", content: "b", path: "/y/1"}, - scanRec{size: 50, head: "a", tail: "a", content: "a", path: "/x/2"}, - scanRec{size: 50, head: "a", tail: "a", content: "a", path: "/x/1"}, - scanRec{size: 50, path: "/x/unhashed"}, + scanRec{size: 50, head: b, tail: b, content: b, path: "/y/2"}, + scanRec{size: 50, head: b, tail: b, content: b, path: "/y/1"}, + scanRec{size: 50, head: a, tail: a, content: a, path: "/x/2"}, + scanRec{size: 50, head: a, tail: a, content: a, path: "/x/1"}, ) reversed := slices.Clone(recs) @@ -386,16 +401,19 @@ func TestWarnfEscapes(t *testing.T) { func TestDupeGroups(t *testing.T) { t.Parallel() + h, tl, c := testHash("h"), testHash("t"), testHash("c") + bigH, bigT, bigC := testHash("H"), testHash("T"), testHash("C") + u := testHash("u") recs := []scanRec{ - {size: 100, head: "h", tail: "t", content: "c", path: "/z/b"}, - {size: 100, head: "h", tail: "t", content: "c", path: "/z/a"}, - {size: 100, head: "h", tail: "t", content: "c", path: "/z/c"}, - {size: 4000, head: "H", tail: "T", content: "C", path: "/big/2"}, - {size: 4000, head: "H", tail: "T", content: "C", path: "/big/1"}, + {size: 100, head: h, tail: tl, content: c, path: "/z/b"}, + {size: 100, head: h, tail: tl, content: c, path: "/z/a"}, + {size: 100, head: h, tail: tl, content: c, path: "/z/c"}, + {size: 4000, head: bigH, tail: bigT, content: bigC, path: "/big/2"}, + {size: 4000, head: bigH, tail: bigT, content: bigC, path: "/big/1"}, // Same size as the /z group but a different head hash. - {size: 100, head: "other", tail: "t", content: "c", path: "/z/d"}, + {size: 100, head: testHash("other"), tail: tl, content: c, path: "/z/d"}, // A singleton signature must not form a group. - {size: 7, head: "u", tail: "u", content: "u", path: "/lonely"}, + {size: 7, head: u, tail: u, content: u, path: "/lonely"}, } groups := dupeGroupsOf(t, recs) @@ -419,16 +437,13 @@ func TestDupeGroups(t *testing.T) { func TestDupeGroupsContentSeparates(t *testing.T) { t.Parallel() - // Same size, head, and tail, but different content hashes: the final - // rung keeps them apart, so no group forms. Matching content groups. - // Records without a content hash never group, not even with each - // other. + // Same size, head, and tail: /b differs from the others only in its + // content hash, so it joins no group, while /a and /c group. + h, tl := testHash("h"), testHash("t") recs := []scanRec{ - {size: 100, head: "h", tail: "t", content: "c1", path: "/a"}, - {size: 100, head: "h", tail: "t", content: "c2", path: "/b"}, - {size: 100, head: "h", tail: "t", content: "c1", path: "/c"}, - {size: 100, head: "h", tail: "t", path: "/d"}, - {size: 100, head: "h", tail: "t", path: "/e"}, + {size: 100, head: h, tail: tl, content: testHash("c1"), path: "/a"}, + {size: 100, head: h, tail: tl, content: testHash("c2"), path: "/b"}, + {size: 100, head: h, tail: tl, content: testHash("c1"), path: "/c"}, } groups := dupeGroupsOf(t, recs) @@ -447,14 +462,15 @@ func TestDupeGroupsMtimeExcluded(t *testing.T) { // mtime is informational only; records differing only in mtime // still group together. + h, tl, c := testHash("h"), testHash("t"), testHash("c") recs := []scanRec{ { - size: 9, mtime: time.Unix(100, 0), head: "h", tail: "t", - content: "c", path: "/m/1", + size: 9, mtime: time.Unix(100, 0), head: h, tail: tl, + content: c, path: "/m/1", }, { - size: 9, mtime: time.Unix(200, 0), head: "h", tail: "t", - content: "c", path: "/m/2", + size: 9, mtime: time.Unix(200, 0), head: h, tail: tl, + content: c, path: "/m/2", }, } @@ -469,11 +485,12 @@ func TestDupeGroupsTieBreak(t *testing.T) { // The hashes sort opposite to the first paths, so ordering the // groups by hash instead of by first path fails this test. + a, b := testHash("a"), testHash("b") recs := []scanRec{ - {size: 50, head: "a", tail: "a", content: "a", path: "/beta/2"}, - {size: 50, head: "a", tail: "a", content: "a", path: "/beta/1"}, - {size: 50, head: "b", tail: "b", content: "b", path: "/alpha/2"}, - {size: 50, head: "b", tail: "b", content: "b", path: "/alpha/1"}, + {size: 50, head: a, tail: a, content: a, path: "/beta/2"}, + {size: 50, head: a, tail: a, content: a, path: "/beta/1"}, + {size: 50, head: b, tail: b, content: b, path: "/alpha/2"}, + {size: 50, head: b, tail: b, content: b, path: "/alpha/1"}, } groups := dupeGroupsOf(t, recs) @@ -495,15 +512,16 @@ func TestDupeGroupsOrderedByFirstPath(t *testing.T) { // /z, /k, which is neither the order of their first paths nor its // reverse, and must write them ordered by first path. The group of // size 7 comes after them all, though its first path sorts first. + a, b, c, d := testHash("a"), testHash("b"), testHash("c"), testHash("d") recs := []scanRec{ - {size: 50, head: "a", tail: "a", content: "a", path: "/k/2"}, - {size: 50, head: "a", tail: "a", content: "a", path: "/k/1"}, - {size: 50, head: "b", tail: "b", content: "b", path: "/z/1"}, - {size: 50, head: "b", tail: "b", content: "b", path: "/z/2"}, - {size: 50, head: "c", tail: "c", content: "c", path: "/a/2"}, - {size: 50, head: "c", tail: "c", content: "c", path: "/a/1"}, - {size: 7, head: "d", tail: "d", content: "d", path: "/0/2"}, - {size: 7, head: "d", tail: "d", content: "d", path: "/0/1"}, + {size: 50, head: a, tail: a, content: a, path: "/k/2"}, + {size: 50, head: a, tail: a, content: a, path: "/k/1"}, + {size: 50, head: b, tail: b, content: b, path: "/z/1"}, + {size: 50, head: b, tail: b, content: b, path: "/z/2"}, + {size: 50, head: c, tail: c, content: c, path: "/a/2"}, + {size: 50, head: c, tail: c, content: c, path: "/a/1"}, + {size: 7, head: d, tail: d, content: d, path: "/0/2"}, + {size: 7, head: d, tail: d, content: d, path: "/0/1"}, } groups := dupeGroupsOf(t, recs) @@ -522,11 +540,12 @@ func TestDupeGroupsOrderedByFirstPath(t *testing.T) { func TestDupeGroupsDeterministic(t *testing.T) { t.Parallel() + a, b := testHash("a"), testHash("b") recs := []scanRec{ - {size: 1, head: "a", tail: "a", content: "a", path: "/p/1"}, - {size: 1, head: "a", tail: "a", content: "a", path: "/p/2"}, - {size: 2, head: "b", tail: "b", content: "b", path: "/q/1"}, - {size: 2, head: "b", tail: "b", content: "b", path: "/q/2"}, + {size: 1, head: a, tail: a, content: a, path: "/p/1"}, + {size: 1, head: a, tail: a, content: a, path: "/p/2"}, + {size: 2, head: b, tail: b, content: b, path: "/q/1"}, + {size: 2, head: b, tail: b, content: b, path: "/q/2"}, } forward := dupeGroupsOf(t, recs) diff --git a/scan.go b/scan.go index 587276f..3c2ec57 100644 --- a/scan.go +++ b/scan.go @@ -5,7 +5,6 @@ import ( "context" "crypto/sha256" "database/sql" - "encoding/hex" "errors" "fmt" "io" @@ -21,32 +20,21 @@ import ( "time" ) -// The duplicate ladder (see hashSignature and README "Duplicate -// detection"). A same-size candidate below headTailMin is hashed in -// full and compared directly; a larger one is separated first by the -// hashes of its end windows, then by a content hash that is exact below -// wholeFileMax and deliberately sampled at or above it. The hash phase -// reads only the end windows of a larger file; the content phase reads -// it for its content hash only once its size, head, and tail match -// another file's. - -// headTailMin is the size threshold for the end-window gate. A file -// smaller than this is hashed in full directly, with no separate head -// and tail step: its head, tail, and content all carry the whole-file -// hash. A file this size or larger is separated first by its end -// windows. -const headTailMin = 10 * 1024 * 1024 +// The hash phase gives every file it reads three hashes (see +// hashSignature and README "Duplicate detection"): head and tail, of +// its first and last headTailWindow bytes, and content. The content +// hash covers the whole file below wholeFileMax, and only sampled +// windows at or above it, so such a file is never read in full. // headTailWindow is the number of bytes hashed from each end of a file -// at or above headTailMin (the head and tail rungs). Because -// headTailMin is far larger than two windows, the head and tail windows -// never overlap. +// for its head and tail. In a file no larger than this, each window is +// the whole file. const headTailWindow = 64 * 1024 -// wholeFileMax is the size boundary between the two content rungs: a +// wholeFileMax is the size boundary between the two content hashes: a // file strictly smaller than this is content-hashed in full; a file // this size or larger is content-hashed by sampling. -const wholeFileMax = 50 * 1024 * 1024 +const wholeFileMax = 100 * 1024 * 1024 // sampleStride is the spacing between content samples for large files: // one window is read at each gigabyte-aligned offset (0, 1 GiB, ...). @@ -64,25 +52,26 @@ const workQueueDepth = 1024 // has already printed its line, so run prints nothing more. var errInterrupted = errors.New("scan interrupted") -// fileRec carries one statted file between the scan phases. dev and -// ino identify the underlying inode so hard-linked paths can share -// one read; both are zero when the platform exposes no inode. +// fileRec carries one statted file between the scan phases. diskSize is +// the bytes the file takes on disk. dev and ino identify the underlying +// inode so hard-linked paths can share one read; both are zero when the +// platform exposes no inode. type fileRec struct { - path string - size int64 - mtime time.Time - dev uint64 - ino uint64 + path string + size int64 + diskSize int64 + mtime time.Time + dev uint64 + ino uint64 } // fileMeta is the in-memory index entry for one existing database -// record: just enough for change detection, plus whether the record -// carries hashes. Hashes stay on disk; at tens of millions of records -// they would dominate the scan's memory. +// record: what the walk compares the file with. Hashes stay on disk; at +// tens of millions of records they would dominate the scan's memory. type fileMeta struct { - size int64 - mtime time.Time - hashed bool + size int64 + diskSize int64 + mtime time.Time } // scanUnit is what every scan phase counts, as its progress display @@ -90,36 +79,31 @@ type fileMeta struct { const scanUnit = "files" // scanPhase names one phase of a scan with its place among all of them, -// as "[2/5] walk", for the phase's progress display. The list here is +// as "[2/4] walk", for the phase's progress display. The list here is // the one place the phases are counted. func scanPhase(name string) string { - return phaseLabel( - []string{"load", "walk", "hash", "update", "content"}, name) + return phaseLabel([]string{"load", "walk", "hash", "update"}, name) } -// runScan implements the scan subcommand: five sequential phases — +// runScan implements the scan subcommand: four sequential phases — // load (which reads the existing records), walk (which stats each file -// as it is discovered), hash, update, content — that synchronize the -// persistent database with the filesystem state under the PATH -// operands. Only files whose size at least one other file shares are -// ever hashed: a size-unique file cannot be a duplicate. A file of -// headTailMin or more gets its content hash only when its size, head, -// and tail match another file's. Flag parsing and the -// at-least-one-operand check are done by cobra. Before anything else, -// the scan names the database on stderr, saying whether it is new. The -// scan holds the lock on the database for its whole run, so a second -// scan fails before it walks the filesystem or opens the database. -// Errors are returned rather than exiting, so that the deferred close — -// which takes the database out of WAL mode — always runs, and the lock -// is released after it. When ctx is cancelled, as by the SIGINT or -// SIGTERM that interruptContext catches, the scan keeps what it has -// hashed (see syncScan), prints how many files its walk reached, and -// returns errInterrupted. workers must be at least 1; the scan command -// rejects anything less. Before the lock is taken, the scan lists the -// filesystems that are not persistent, on which the walk records -// nothing; unless noDefaultExcludes is set, the walk does not enter the -// default excludes either. Once the database is open, the scan notes -// its files, which the walk passes over. +// as it is discovered), hash, update — that synchronize the persistent +// database with the filesystem state under the PATH operands. Flag +// parsing and the at-least-one-operand check are done by cobra. Before +// anything else, the scan names the database on stderr, saying whether +// it is new. The scan holds the lock on the database for its whole run, +// so a second scan fails before it walks the filesystem or opens the +// database. Errors are returned rather than exiting, so that the +// deferred close — which takes the database out of WAL mode — always +// runs, and the lock is released after it. When ctx is cancelled, as by +// the SIGINT or SIGTERM that interruptContext catches, the scan keeps +// what it has hashed (see syncScan), prints how many files its walk +// reached, and returns errInterrupted. workers must be at least 1; the +// scan command rejects anything less. Before the lock is taken, the +// scan lists the filesystems that are not persistent, on which the walk +// records nothing; unless noDefaultExcludes is set, the walk does not +// enter the default excludes either. Once the database is open, the +// scan notes its files, which the walk passes over. func runScan(ctx context.Context, roots []string, workers int, oneFS, noDefaultExcludes bool, ) error { @@ -225,29 +209,30 @@ func interrupted(walked int) error { return errInterrupted } -// resolveRoots converts each PATH operand to an absolute, lexically -// cleaned path (symlinks are not resolved) and verifies that it -// exists. Database records are keyed by absolute path, so scan results -// must not depend on the working directory. +// resolveRoots converts each PATH operand to its real path: absolute, +// with every symlink in it followed. Database records are keyed by +// that path, so scan results depend neither on the working directory +// nor on the symlinks an operand was given through. An operand that +// cannot be resolved, because it or a symlink's target does not exist +// or cannot be read, is a fatal error before any scanning. func resolveRoots(roots []string) ([]string, error) { - abs := make([]string, 0, len(roots)) + resolved := make([]string, 0, len(roots)) for _, root := range roots { - a, err := filepath.Abs(root) + a, err := absolutePath(root) if err != nil { return nil, fmt.Errorf("resolve %s: %w", root, err) } - // A nonexistent operand is a fatal error before any scanning. - _, err = os.Lstat(a) + r, err := filepath.EvalSymlinks(a) if err != nil { - return nil, err + return nil, fmt.Errorf("resolve %s: %w", root, err) } - abs = append(abs, a) + resolved = append(resolved, r) } - return abs, nil + return resolved, nil } // pruneRoots drops operands already covered by another operand: @@ -360,9 +345,8 @@ type scanStats struct { type scanState struct { db *sql.DB existing map[string]fileMeta - sizes []int64 // size census: every walked file, plus records outside the roots - toHash []fileRec // files whose size is shared: must be read - sentinels []fileRec // new/changed size-unique files: recorded without hashes + toHash []fileRec // new and changed files: must be read + diskSizes []fileRec // unchanged files whose disk_size changed batch []scanRec st scanStats } @@ -391,25 +375,21 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string, } // runPhases synchronizes the database with the filesystem under roots -// in five sequential phases: load (read the existing records), walk -// (enumerate and stat every file, building a complete size census), -// hash (read only the new or changed — or previously unhashed — files -// whose size at least one other file shares, committing results in -// batches as they arrive), update (record the size-unique files without -// reading them, and delete the records the scan no longer verifies), -// and content (fill in the content hash of every record of headTailMin -// or more whose size, head, and tail match another record's). Records -// outside the roots are never touched, except that the content phase -// fills in their content hash. Operands the walk cannot start from, or -// that are not on a persistent filesystem, are dropped first, so the -// records beneath them are treated as if those operands had not been -// given. Below an operand, the walk follows rules. +// in four sequential phases: load (read the existing records), walk +// (enumerate and stat every file), hash (read only the new or changed +// files, committing results in batches as they arrive), and update +// (delete the records the scan no longer verifies). Records outside the +// roots are never touched. Operands the walk cannot start from, or that +// are not on a persistent filesystem, are dropped first, so the records +// beneath them are treated as if those operands had not been given. +// Below an operand, the walk follows rules. func (s *scanState) runPhases(ctx context.Context, roots []string, workers int, oneFS bool, rules walkRules, ) error { // Types and filesystems are checked before pruning so that an // operand under a dropped one is still scanned, not dropped as lying - // under it. + // under it, and so that an operand inside a .zfs directory gets its + // warning even when it lies under another operand. roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules, deviceOfPath) @@ -418,41 +398,36 @@ func (s *scanState) runPhases(ctx context.Context, roots []string, return err } - changed, unhashed := s.walkPhase( - startWalk(ctx, roots, oneFS, rules, workers), roots) + s.walkPhase(startWalk(ctx, roots, oneFS, rules, workers), roots) - // A cancelled walk stops early, so its size census covers only part - // of the roots, and every file it never reached would look vanished - // to the update phase. Stop before anything is written or deleted. + // A cancelled walk stops early, so every file it never reached would + // look vanished to the update phase. Stop before anything is written + // or deleted. err = ctx.Err() if err != nil { return err } - s.partition(changed, unhashed) - err = s.hashPhase(ctx, workers) if err != nil { return err } - err = s.updatePhase(ctx) - if err != nil { - return err - } - - return s.contentPhase(ctx, workers, rules) + return s.updatePhase(ctx) } // walkableRoots returns the operands the walk can start from: regular -// files, and directories not named .zfs. Every other operand is warned -// about, counted as skipped, and dropped. A dropped operand is no -// longer a root, so the records stored beneath it are treated as if it -// had not been given: those under no other root count as outside the -// roots and are not deleted as unverified. An operand on a filesystem -// in rules that is not persistent is dropped the same way, with a line -// saying so that is not a warning, and is not counted as skipped. An -// operand that fails lstat here is kept, and the walk warns about it. +// files, and directories not named .zfs, none of them inside a +// directory named .zfs. The operands are real paths from resolveRoots, +// so these checks apply to what a symlink operand points at. Every +// other operand is warned about, counted as skipped, and dropped. A +// dropped operand is no longer a root, so the records stored beneath it +// are treated as if it had not been given: those under no other root +// count as outside the roots and are not deleted as unverified. An +// operand on a filesystem in rules that is not persistent is dropped +// the same way, with a line saying so that is not a warning, and is not +// counted as skipped. An operand that fails lstat here is kept, and the +// walk warns about it unless it no longer exists. func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { kept := make([]string, 0, len(roots)) @@ -483,14 +458,12 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { } // loadIndex indexes the database records under the scan roots for -// change detection and collects the sizes of every record outside -// them: out-of-scope records join the size census so a scanned file -// can be recognized as a possible duplicate of a tree scanned -// separately into the same database. +// change detection. It reads every record, those outside the roots +// included, so its display counts against the number of records. func (s *scanState) loadIndex(ctx context.Context, roots []string) error { // Indexing tens of millions of records takes real time; without a - // display the scan looks hung before the walk begins. Counting them - // first gives the display its total. + // display the scan looks hung before the walk begins. The number of + // records, read first, gives the display its total. prog := newProgress(progressParams{ label: scanPhase("load"), unit: scanUnit, total: -1, waitingFor: countingRecords, stalledOn: "db", @@ -506,37 +479,25 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error { s.existing = make(map[string]fileMeta) - return loadFileMeta(ctx, s.db, - func(path string, size int64, mtime time.Time, hashed bool) { - prog.increment() + return loadFileMeta(ctx, s.db, func(path string, m fileMeta) { + prog.increment() - if underAnyRoot(path, roots) { - s.existing[path] = fileMeta{ - size: size, mtime: mtime, hashed: hashed, - } - - return - } - - s.sizes = append(s.sizes, size) - }) + if underAnyRoot(path, roots) { + s.existing[path] = m + } + }) } -// walkPhase drains the walk, appending every walked file's size to -// the census and resolving what it can immediately: an unchanged file -// whose record already has hashes needs nothing from the hash phase -// (the content phase may still fill in its content hash). It returns -// the new-or-changed files and the unchanged files whose records lack -// hashes; both remain candidates until the census decides whether -// their sizes are shared. A record at or below a path the walk left -// out, because -x stopped at it, it is a default exclude, or it is not -// on a persistent filesystem, counts as outside the roots: it is not -// deleted, and its size joins the census. That does not hold below a -// root at or inside that path, since that root's own walk verifies the +// walkPhase drains the walk and queues every new or changed file for +// the hash phase. An unchanged file is not read, and its record counts +// as verified. When its disk_size differs from the record's, it is +// queued for the update phase, which writes the new disk_size. A record +// at or below a path the walk left out, because -x stopped at it, it is +// a default exclude, or it is not on a persistent filesystem, counts as +// outside the roots: it is not deleted. That does not hold below a root +// at or inside that path, since that root's own walk verifies the // records below it; nor is a line printed for such a root. -func (s *scanState) walkPhase( - events <-chan walkEvent, roots []string, -) ([]fileRec, []fileRec) { +func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) { // The records the last scan left under the roots estimate how many // files the walk will find. A first scan has none, and no estimate. estimate := int64(len(s.existing)) @@ -549,10 +510,7 @@ func (s *scanState) walkPhase( estimate: true, stalledOn: "disk", }) - var ( - changed, unhashed []fileRec - leftOut []string - ) + var leftOut []string for ev := range events { if ev.otherFSDir != "" { @@ -582,28 +540,32 @@ func (s *scanState) walkPhase( continue } - s.sizes = append(s.sizes, ev.rec.size) s.st.walked++ prog.increment() old, ok := s.existing[ev.rec.path] + if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) { + s.toHash = append(s.toHash, ev.rec) - switch { - case !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime): - changed = append(changed, ev.rec) - case old.hashed: - delete(s.existing, ev.rec.path) + continue + } - s.st.unchanged++ - default: - unhashed = append(unhashed, ev.rec) + delete(s.existing, ev.rec.path) + + s.st.unchanged++ + + // The space a file takes on disk can change while its size and + // mtime stay the same. ZFS, for one, counts a new file's blocks + // up to seconds after the write. + if old.diskSize != ev.rec.diskSize { + s.diskSizes = append(s.diskSizes, ev.rec) } } prog.finish() - for path, old := range s.existing { + for path := range s.existing { // Kept only when a path the walk left out lies below the deepest // root above the record, or is the record's own. A root at or // below that path was walked on its own and did not find the @@ -612,56 +574,8 @@ func (s *scanState) walkPhase( left := deepestRoot(path, leftOut) if len(left) > len(deepestRoot(path, roots)) { delete(s.existing, path) - - s.sizes = append(s.sizes, old.size) } } - - return changed, unhashed -} - -// partition decides each candidate's disposition now that the size -// census is complete. A file whose size no other file shares cannot -// be a duplicate and is never read: a new or changed one is recorded -// without hashes, an unchanged unhashed one keeps its record. Every -// file with a shared size queues for the hash phase. -func (s *scanState) partition(changed, unhashed []fileRec) { - slices.Sort(s.sizes) - - for _, rec := range changed { - if s.sizeShared(rec.size) { - s.toHash = append(s.toHash, rec) - - continue - } - - s.sentinels = append(s.sentinels, rec) - s.resolve(rec.path) - } - - for _, rec := range unhashed { - if s.sizeShared(rec.size) { - s.toHash = append(s.toHash, rec) - - continue - } - - delete(s.existing, rec.path) - - s.st.unchanged++ - } - - s.sizes = nil -} - -// sizeShared reports whether at least two census entries have this -// size. Every candidate's own size is in the census exactly once, so -// a second entry means another file (or a record outside the scan -// roots) could share its content. -func (s *scanState) sizeShared(size int64) bool { - i, found := slices.BinarySearch(s.sizes, size) - - return found && i+1 < len(s.sizes) && s.sizes[i+1] == size } // resolve counts one written record as added or updated and marks its @@ -725,44 +639,31 @@ func isDatabaseFile(f fileRec, rules walkRules) bool { }) } -// hashPhase hashes every queued file with hashSignature — the head and -// tail of a file of headTailMin or more, the whole file below that — -// committing completed records to the database in batches as results -// arrive, so a long scan persists its progress as it goes (an -// interrupted scan resumes cheaply: the next run skips everything +// hashPhase hashes every queued file with hashSignature, one read per +// inode run, committing completed records to the database in batches +// as results arrive, so a long scan persists its progress as it goes +// (an interrupted scan resumes cheaply: the next run skips everything // already recorded). A run that fails to hash is warned about and -// skipped; stale records for its paths, if any, are deleted by the -// update phase. -func (s *scanState) hashPhase(ctx context.Context, workers int) error { - runs := hashRuns(s.toHash) - s.toHash = nil - - return s.readRuns(ctx, workers, "hash", runs, hashSignature, s.recordRun) -} - -// readRuns reads runs with the worker pool, one read per inode run, in -// the order given, under the progress display of the phase named label. -// The workers compute each run's hashes with hash, and each result goes -// to record; a run that fails to read is warned about and counted as -// skipped instead. The total counts actual reads, so the display shows -// a real ETA. +// counted as skipped; stale records for its paths, if any, are deleted +// by the update phase. A run whose file no longer exists gets no record +// and no warning, and is not counted as skipped. The total counts actual +// reads, so the display's time remaining is meaningful. // // Returning early — a failed database write, or a cancelled scan — must // not strand the pool: the feeder would park forever on a full jobs // channel and every worker on a full results channel. The deferred stop // is what prevents that. -func (s *scanState) readRuns(ctx context.Context, workers int, - label string, runs [][]fileRec, - hash func(path string, size int64) (string, string, string, error), - record func(ctx context.Context, r hashResult) error, -) error { - pool := startHashPool(ctx, runs, workers, hash) +func (s *scanState) hashPhase(ctx context.Context, workers int) error { + runs := hashRuns(s.toHash) + s.toHash = nil + + pool := startHashPool(ctx, runs, workers, hashSignature) defer pool.stop() // Between results the phase may wait on a file being read or on a // batch of records being written. prog := newProgress(progressParams{ - label: scanPhase(label), unit: scanUnit, total: int64(len(runs)), + label: scanPhase("hash"), unit: scanUnit, total: int64(len(runs)), stalledOn: "disk/db", }) defer prog.finish() @@ -778,15 +679,20 @@ func (s *scanState) readRuns(ctx context.Context, workers int, prog.increment() + // A file removed since the walk is no error: it gets no record. + if errors.Is(r.err, fs.ErrNotExist) { + continue + } + if r.err != nil { s.st.skipped += len(r.run) - prog.warnf("%s %s: %v", label, r.run[0].path, r.err) + prog.warnf("hash %s: %v", r.run[0].path, r.err) continue } - err := record(ctx, r) + err := s.recordRun(ctx, r) if err != nil { return err } @@ -803,12 +709,13 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error { s.resolve(rec.path) s.batch = append(s.batch, scanRec{ - size: rec.size, - mtime: rec.mtime, - head: r.head, - tail: r.tail, - content: r.content, - path: rec.path, + size: rec.size, + diskSize: rec.diskSize, + mtime: rec.mtime, + head: r.head, + tail: r.tail, + content: r.content, + path: rec.path, }) } @@ -824,7 +731,7 @@ func (s *scanState) commitFullBatch(ctx context.Context) error { return nil } - err := applyBatch(ctx, s.db, s.batch, nil, nil) + err := applyBatch(ctx, s.db, s.batch, nil, nil, nil) if err != nil { return err } @@ -835,10 +742,10 @@ func (s *scanState) commitFullBatch(ctx context.Context) error { } // updatePhase writes the scan's tail under one progress display: the -// final partial batch of hashed records, a hash-less record for every -// size-unique new or changed file, and deletions for every record the -// scan did not verify (vanished files, plus paths that failed to stat -// or hash). +// final partial batch of hashed records, the new disk_size of every +// unchanged file whose disk_size changed, and deletions for every +// record the scan did not verify (vanished files, plus paths that +// failed to stat or hash). func (s *scanState) updatePhase(ctx context.Context) error { deletes := make([]string, 0, len(s.existing)) for path := range s.existing { @@ -850,7 +757,7 @@ func (s *scanState) updatePhase(ctx context.Context) error { s.st.removed = len(deletes) - total := len(s.batch) + len(s.sentinels) + len(deletes) + total := len(s.batch) + len(s.diskSizes) + len(deletes) prog := newProgress(progressParams{ label: scanPhase("update"), unit: scanUnit, total: int64(total), stalledOn: "db", @@ -865,209 +772,14 @@ func (s *scanState) updatePhase(ctx context.Context) error { s.batch = nil - // Sentinel records are converted in batch-sized chunks rather than - // materialized all at once; a first scan can have millions. - for chunk := range slices.Chunk(s.sentinels, updateBatchSize) { - recs := make([]scanRec, 0, len(chunk)) - for _, rec := range chunk { - recs = append(recs, scanRec{ - size: rec.size, mtime: rec.mtime, path: rec.path, - }) - } - - err = applyBatch(ctx, s.db, recs, nil, prog) - if err != nil { - return err - } + err = applyDiskSizes(ctx, s.db, s.diskSizes, prog) + if err != nil { + return err } return applyChanges(ctx, s.db, nil, deletes, prog) } -// contentPhase fills in the content hash of every record of headTailMin -// or more that lacks one and whose size, head, and tail equal another -// record's, anywhere in the database: records from this scan and -// records stored by earlier scans, inside or outside the roots. Only -// such a file can still be a duplicate, so no other file of headTailMin -// or more is read beyond its end windows. The files are read with the -// hash phase's worker pool and their records written back in batches. A -// failed read is warned about and counted as skipped; the record keeps -// its empty content, so it is never grouped, and a later scan tries -// again. No file on a filesystem in rules that is not persistent is -// read. -func (s *scanState) contentPhase(ctx context.Context, workers int, - rules walkRules, -) error { - toRead, recs, err := s.contentCandidates(ctx, rules) - if err != nil { - return err - } - - err = s.readRuns(ctx, workers, "content", hashRuns(toRead), - hashContentOnly, func(ctx context.Context, r hashResult) error { - // Every path in the run keeps its record's head and tail - // and gains the one content hash read for the run. - for _, f := range r.run { - rec := recs[f.path] - rec.content = r.content - s.batch = append(s.batch, rec) - } - - return s.commitFullBatch(ctx) - }) - if err != nil { - return err - } - - // The last partial batch is written under a display of its own, as - // the update phase writes the hash phase's. - prog := newProgress(progressParams{ - label: scanPhase("content"), unit: scanUnit, - total: int64(len(s.batch)), stalledOn: "db", - }) - defer prog.finish() - - return applyChanges(ctx, s.db, s.batch, nil, prog) -} - -// contentCandidates returns the files the content phase reads, and -// their records by path. Every record contentCandidatesSQL returns has -// its file checked with lstat, whether or not it already has a content -// hash: a file that is gone, is no longer a regular file, has changed -// by the walk's rule, or is on a filesystem in rules that is not -// persistent keeps its record as it is and does not count as a match -// for the others, and any other lstat error is warned about and counted -// as skipped, with the same result. If such a record has no content -// hash, it stays out of duplicate groups; if it has one, it is still -// reported until a scan covering its own tree updates or removes it. -// The files of a group that pass and have no content hash are read only -// if at least minGroupSize of the group's files pass, so a group whose -// other members are all stale costs no reads. Only the records to be -// read are kept. -func (s *scanState) contentCandidates( - ctx context.Context, rules walkRules, -) ([]fileRec, map[string]scanRec, error) { - // The search and the checks take real time on a large database; - // without a display the scan looks hung before the reads begin. - // The display says it is searching until SQLite returns the first - // record. The checks start there, against a count of the records - // the search returns, and so do their rate and ETA. Between records - // they may wait on a file's lstat or on the search. - prog := newProgress(progressParams{ - label: scanPhase("content"), unit: scanUnit, total: -1, - waitingFor: "searching the database", stalledOn: "disk/db", - }) - defer prog.finish() - - total, err := countContentCandidates(ctx, s.db) - if err != nil { - return nil, nil, err - } - - var ( - searching = true // no record has been returned yet - toRead []fileRec - first scanRec // the current group's first record - passed int // the current group's files that passed the check - unread []fileRec // those of them without a content hash - ) - - recs := make(map[string]scanRec) - - // endGroup queues the current group's files to read if at least - // minGroupSize of its files passed, and drops their records if not. - endGroup := func() { - if passed >= minGroupSize { - toRead = append(toRead, unread...) - } else { - for _, f := range unread { - delete(recs, f.path) - } - } - - passed, unread = 0, nil - } - - err = loadContentCandidates(ctx, s.db, func(r scanRec, hashed bool) { - if searching { - searching = false - - prog.setTotal(total) - } - - prog.increment() - - if r.size != first.size || r.head != first.head || r.tail != first.tail { - endGroup() - - first = r - } - - f, ok, err := unchangedFile(r, rules) - if err != nil { - s.st.skipped++ - - prog.warnf("content %s: %v", r.path, err) - } - - if !ok { - return - } - - passed++ - - if !hashed { - unread = append(unread, f) - recs[r.path] = r - } - }) - if err != nil { - return nil, nil, err - } - - // A search that returned no record still ends showing its count. - if searching { - prog.setTotal(total) - } - - endGroup() - - return toRead, recs, nil -} - -// unchangedFile lstats the file r names and returns it for reading if -// it is still the regular file r records: the same size, and an mtime -// no newer than recorded (the walk's change rule). A file that is gone, -// has changed, or is on a filesystem in rules that is not persistent -// reports false; any other lstat error is returned. -func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) { - fi, err := os.Lstat(r.path) - if errors.Is(err, fs.ErrNotExist) { - return fileRec{}, false, nil - } - - if err != nil { - return fileRec{}, false, err - } - - if !fi.Mode().IsRegular() || fi.Size() != r.size || - mtimeAfter(fi.ModTime(), r.mtime) { - return fileRec{}, false, nil - } - - dev, ino := inodeOfInfo(fi) - - // The scan reads nothing on a filesystem that is not persistent, - // whatever record an earlier scan left for the file. - if _, found := rules.nonPersistent[dev]; found { - return fileRec{}, false, nil - } - - return fileRec{ - path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino, - }, true, nil -} - // mtimeAfter reports whether mtime a is later than mtime b. // Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it. func mtimeAfter(a, b time.Time) bool { @@ -1146,10 +858,12 @@ type walkEvent struct { // startWalk seeds every root into the shared walk worker pool and // returns the event stream: one record per regular file, one warning // event per per-path error, and one event per path -x or rules kept the -// walk from entering or recording. The channel is closed when the walk -// completes, and also when ctx is cancelled — every goroutine in the -// pool abandons its blocking send in that case, so the consumer sees a -// truncated but properly terminated stream instead of a stalled one. +// walk from entering or recording. A path that no longer exists when +// the walk reaches it is no error, and gets no event. The channel is +// closed when the walk completes, and also when ctx is cancelled — +// every goroutine in the pool abandons its blocking send in that case, +// so the consumer sees a truncated but properly terminated stream +// instead of a stalled one. func startWalk(ctx context.Context, roots []string, oneFS bool, rules walkRules, workers int, ) <-chan walkEvent { @@ -1182,9 +896,18 @@ func sendEvent(ctx context.Context, events chan<- walkEvent, // operandWarning returns the one-line warning for an operand the walk // does not start from, naming the path and what it is, or "" for one it -// does: a regular file, or a directory not named .zfs. Symlinks are -// never followed, including as operands. +// does: a regular file, or a directory not named .zfs, that is not +// inside a directory named .zfs. resolveRoots has already followed the +// operand's symlinks, so a symlink here is an operand replaced by one +// after it was resolved, and is not followed. func operandWarning(root string, fi fs.FileInfo) string { + // The walk never enters a .zfs directory, so it starts from nothing + // inside one either. + if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") { + return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory", + root) + } + var kind string switch mode := fi.Mode(); { @@ -1216,13 +939,19 @@ func operandWarning(root string, fi fs.FileInfo) string { // one of the scan's own database files in rules, and a directory // operand becomes an initial job. walkableRoots has already dropped // every other operand. One that has changed into something else since -// is warned about and skipped here; it is still a root, so the records -// stored beneath it, other than those under another root, are deleted -// as unverified. +// is warned about and skipped here, and one removed since is skipped +// without a warning. Either is still a root, so the records stored +// beneath it, other than those under another root, are deleted as +// unverified. func seedRoot(ctx context.Context, root string, rules walkRules, events chan<- walkEvent, ) []dirJob { fi, err := os.Lstat(root) + if errors.Is(err, fs.ErrNotExist) { + // Removed since the scan started: no error, and no files. + return nil + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", root, err), @@ -1247,11 +976,12 @@ func seedRoot(ctx context.Context, root string, rules walkRules, dev, ino := inodeOfInfo(fi) rec := fileRec{ - path: root, - size: fi.Size(), - mtime: fi.ModTime(), - dev: dev, - ino: ino, + path: root, + size: fi.Size(), + diskSize: diskSizeOfInfo(fi), + mtime: fi.ModTime(), + dev: dev, + ino: ino, } if !isDatabaseFile(rec, rules) { @@ -1340,11 +1070,17 @@ func dispatchDirs(ctx context.Context, initial []dirJob, // walkOneDir reads one directory, emitting an event per regular-file // entry and a warning event per unreadable one, and returns the -// subdirectories to descend into. +// subdirectories to descend into. A directory or entry removed since +// the walk found it gets no event. func walkOneDir(ctx context.Context, job dirJob, oneFS bool, rules walkRules, events chan<- walkEvent, ) []dirJob { entries, err := os.ReadDir(job.path) + if errors.Is(err, fs.ErrNotExist) { + // Removed since its parent was read: no error, and no files. + return nil + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", job.path, err), @@ -1387,16 +1123,22 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool, // emitFile stats one regular directory entry and emits its record. // The lstat happens here in the walk worker, while the directory's -// metadata is still hot; a path that fails to stat (or stops being a -// regular file) between the directory read and the lstat is warned -// about and skipped. A file on a filesystem in rules that is not -// persistent, as through a bind mount of one file, is not recorded: it -// gets an event naming it instead. One of the scan's own database files -// in rules gets no event at all. +// metadata is still hot; a path that fails to stat between the +// directory read and the lstat is warned about and skipped, unless it +// no longer exists, and one that has stopped being a regular file is +// skipped. A file on a filesystem in rules that is not persistent, as +// through a bind mount of one file, is not recorded: it gets an event +// naming it instead. One of the scan's own database files in rules gets +// no event at all. func emitFile(ctx context.Context, p string, e fs.DirEntry, rules walkRules, events chan<- walkEvent, ) { info, err := e.Info() + if errors.Is(err, fs.ErrNotExist) { + // Removed since its directory was read: no error. + return + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("stat %s: %v", p, err), @@ -1419,11 +1161,12 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry, dev, ino := inodeOfInfo(info) rec := fileRec{ - path: p, - size: info.Size(), - mtime: info.ModTime(), - dev: dev, - ino: ino, + path: p, + size: info.Size(), + diskSize: diskSizeOfInfo(info), + mtime: info.ModTime(), + dev: dev, + ino: ino, } if !isDatabaseFile(rec, rules) { @@ -1448,6 +1191,11 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry, } info, err := e.Info() + if errors.Is(err, fs.ErrNotExist) { + // Removed since its parent was read: no error, and no files. + return dirJob{}, false + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", p, err), @@ -1510,15 +1258,31 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) { return statDev(st), st.Ino } +// statBlockSize is the unit of st_blocks in an lstat result on Linux and +// macOS, whatever the filesystem's own block size. +const statBlockSize = 512 + +// diskSizeOfInfo returns the bytes a file takes on disk: its allocated +// blocks, st_blocks × 512. On a compressed or sparse file it is less +// than the file's size. It is 0 when the platform exposes no block +// count. +func diskSizeOfInfo(fi fs.FileInfo) int64 { + st, ok := fi.Sys().(*syscall.Stat_t) + if !ok { + return 0 + } + + return st.Blocks * statBlockSize +} + // hashResult carries the hashes computed for one inode run (or the // error that prevented computing them) from the pool's workers to the -// phase that started the pool: head, tail, and content from -// hashSignature, content alone from hashContentOnly. +// hash phase. Its run leaves out the paths found removed. type hashResult struct { run []fileRec - head string - tail string - content string + head [sha256.Size]byte + tail [sha256.Size]byte + content [sha256.Size]byte err error } @@ -1538,7 +1302,8 @@ type hashPool struct { // hash each run's first path with hash (all paths in a run are hard // links to the same inode) and write one result per run. func startHashPool(ctx context.Context, runs [][]fileRec, workers int, - hash func(path string, size int64) (string, string, string, error), + hash func(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error), ) *hashPool { ctx, cancel := context.WithCancel(ctx) @@ -1603,7 +1368,8 @@ func feedHashJobs(ctx context.Context, runs [][]fileRec, // the hash of only delays that. func hashWorker(ctx context.Context, jobs <-chan []fileRec, results chan<- hashResult, - hash func(path string, size int64) (string, string, string, error), + hash func(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error), ) { for run := range jobs { if ctx.Err() != nil { @@ -1612,6 +1378,13 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec, head, tail, content, err := hash(run[0].path, run[0].size) + // A path removed since the walk is dropped from its run. The + // next path is a hard link to the same file, and is read instead. + for errors.Is(err, fs.ErrNotExist) && len(run) > 1 { + run = run[1:] + head, tail, content, err = hash(run[0].path, run[0].size) + } + select { case results <- hashResult{ run: run, head: head, tail: tail, content: content, err: err, @@ -1654,100 +1427,73 @@ func openRegularFile(path string) (*os.File, error) { return f, nil } -// emptyHash is the lowercase-hex SHA-256 of the empty input: the head, -// tail, and content hash of every zero-length file. -const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" + - "27ae41e4649b934ca495991b7852b855" +// hashSignature computes the three hashes the hash phase records for a +// file: head and tail (see hashEnds) and content (see hashContent). +// With the file size they form its duplicate signature, and two files +// are duplicates only when all four agree. size is the value recorded +// when the file was statted; a zero-length file has constant hashes and +// is never opened. On an error every hash is left zero. +func hashSignature(path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, error, +) { + var zero [sha256.Size]byte -// hashSignature computes the hashes the hash phase records for a file -// whose size is shared; with the file size they form its duplicate -// signature. A file below headTailMin is hashed in full and its -// whole-file SHA-256 is returned as head, tail, and content alike — -// that range takes no separate end-window step. For a file at or above -// headTailMin only the head and tail are computed, the SHA-256 of its -// first and last headTailWindow bytes, and content is returned empty: -// the content phase computes it with hashContentOnly once the file's -// size, head, and tail match another file's. Two files are duplicates -// only when all four agree; any mismatch means not a duplicate. size -// is the value recorded when the file was statted; a zero-length file -// has constant hashes and is never opened. -func hashSignature(path string, size int64) (string, string, string, error) { if size == 0 { - return emptyHash, emptyHash, emptyHash, nil + empty := sha256.Sum256(nil) + + return empty, empty, empty, nil } f, err := openRegularFile(path) if err != nil { - return "", "", "", err + return zero, zero, zero, err } defer func() { _ = f.Close() }() - // Below the threshold the whole file is hashed directly, with no - // end-window step: head and tail both carry the whole-file hash. - if size < int64(headTailMin) { - content, err := hashWhole(f, size) - if err != nil { - return "", "", "", err - } - - return content, content, content, nil - } - head, tail, err := hashEnds(f, size) if err != nil { - return "", "", "", err + return zero, zero, zero, err } - return head, tail, "", nil -} - -// hashContentOnly returns the content hash of the file at path, which -// is at least headTailMin bytes: the content phase's read. head and -// tail are returned empty, because the content phase keeps the ones its -// records already hold. -func hashContentOnly(path string, size int64) (string, string, string, error) { - f, err := openRegularFile(path) - if err != nil { - return "", "", "", err - } - - defer func() { _ = f.Close() }() - content, err := hashContent(f, size) + if err != nil { + return zero, zero, zero, err + } - return "", "", content, err + return head, tail, content, nil } // hashEnds returns the SHA-256 of the first and last headTailWindow -// bytes of f. It is called only for files at least headTailMin, which -// is far larger than two windows, so the windows never overlap and both -// reads are always full. -func hashEnds(f *os.File, size int64) (string, string, error) { - buf := make([]byte, headTailWindow) +// bytes of f. In a file no larger than headTailWindow, each window is +// the whole file. Reading fewer bytes than asked means the file shrank +// between the stat and the hash, and is an error. +func hashEnds(f *os.File, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, error, +) { + var zero [sha256.Size]byte + + buf := make([]byte, min(int64(headTailWindow), size)) _, err := f.ReadAt(buf, 0) if err != nil { - return "", "", err + return zero, zero, err } - h := sha256.Sum256(buf) - head := hex.EncodeToString(h[:]) + head := sha256.Sum256(buf) - _, err = f.ReadAt(buf, size-int64(headTailWindow)) + _, err = f.ReadAt(buf, size-int64(len(buf))) if err != nil { - return "", "", err + return zero, zero, err } - t := sha256.Sum256(buf) - - return head, hex.EncodeToString(t[:]), nil + return head, sha256.Sum256(buf), nil } -// hashContent returns the content-rung hash of f: the SHA-256 of the -// whole file when it is smaller than wholeFileMax, or of sampled -// windows when it is that size or larger. -func hashContent(f *os.File, size int64) (string, error) { +// hashContent returns the content hash of f: the SHA-256 of the whole +// file when it is smaller than wholeFileMax, or of sampled windows when +// it is that size or larger. +func hashContent(f *os.File, size int64) ([sha256.Size]byte, error) { if size >= int64(wholeFileMax) { return hashSamples(f, size) } @@ -1760,30 +1506,30 @@ func hashContent(f *os.File, size int64) (string, error) { // reads. Reading fewer than size bytes means the file shrank between // the stat and the hash; that is an error rather than a hash of content // that no longer matches the recorded size. -func hashWhole(f *os.File, size int64) (string, error) { +func hashWhole(f *os.File, size int64) ([sha256.Size]byte, error) { h := sha256.New() n, err := io.Copy(h, io.NewSectionReader(f, 0, size)) if err != nil { - return "", err + return [sha256.Size]byte{}, err } if n != size { - return "", fmt.Errorf("read %d of %d bytes: %w", n, size, - io.ErrUnexpectedEOF) + return [sha256.Size]byte{}, fmt.Errorf("read %d of %d bytes: %w", + n, size, io.ErrUnexpectedEOF) } - return hex.EncodeToString(h.Sum(nil)), nil + return [sha256.Size]byte(h.Sum(nil)), nil } // hashSamples feeds sampleWindow bytes at each gigabyte-aligned offset // (0, sampleStride, 2*sampleStride, ... while inside the file), in -// order, into one hash, each window truncated at end of file. This is -// the probabilistic large-file rung: two files of equal size agreeing -// on every sample are reported as duplicates without every byte being -// read. Because size is part of the signature, files of different sizes -// never reach this comparison, so the sample boundaries always align. -func hashSamples(f *os.File, size int64) (string, error) { +// order, into one hash, each window truncated at end of file. Two files +// of equal size agreeing on every sample are reported as duplicates +// without every byte being read. Because size is part of the signature, +// only files of one size are compared, so their sample boundaries +// always align. +func hashSamples(f *os.File, size int64) ([sha256.Size]byte, error) { h := sha256.New() buf := make([]byte, sampleWindow) @@ -1792,11 +1538,11 @@ func hashSamples(f *os.File, size int64) (string, error) { _, err := f.ReadAt(buf[:n], off) if err != nil { - return "", err + return [sha256.Size]byte{}, err } h.Write(buf[:n]) } - return hex.EncodeToString(h.Sum(nil)), nil + return [sha256.Size]byte(h.Sum(nil)), nil } diff --git a/scan_test.go b/scan_test.go index 1d719ef..298b248 100644 --- a/scan_test.go +++ b/scan_test.go @@ -5,7 +5,6 @@ import ( "context" "crypto/sha256" "database/sql" - "encoding/hex" "errors" "fmt" "io" @@ -43,11 +42,28 @@ func writeFile(t *testing.T, dir, name string, data []byte) string { return p } -// hexSum returns the lowercase-hex SHA-256 of data. -func hexSum(data []byte) string { - s := sha256.Sum256(data) +// diskSizeOf returns the bytes the file at path takes on disk, from its +// own lstat: st_blocks counts 512-byte units. ZFS counts a write's +// blocks only when it commits the write, up to seconds later, and one +// commit can take the file through several growing sizes. So this value +// can change at any moment. Neither fsync nor unix.Sync hastens that, +// but a unix.Sync call often takes long enough that the blocks are +// counted when it returns. A test that needs the value to change waits +// for the change. A test that compares it with a scan's disk_size reads +// it before and after the scan. The rewrite test rescans until the two +// reads match, then compares exactly. The other two tests fail only on +// a disk_size outside the two reads. +func diskSizeOf(t *testing.T, path string) int64 { + t.Helper() - return hex.EncodeToString(s[:]) + var st syscall.Stat_t + + err := syscall.Lstat(path, &st) + if err != nil { + t.Fatal(err) + } + + return st.Blocks * 512 } // pattern returns n bytes of deterministic content seeded by tag. @@ -62,7 +78,9 @@ func pattern(tag byte, n int) []byte { // sig returns a file's full signature (head, tail, content), failing the // test on any error. -func sig(t *testing.T, path string, size int64) (string, string, string) { +func sig(t *testing.T, path string, size int64) ( + [sha256.Size]byte, [sha256.Size]byte, [sha256.Size]byte, +) { t.Helper() head, tail, content, err := hashSignature(path, size) @@ -73,10 +91,12 @@ func sig(t *testing.T, path string, size int64) (string, string, string) { return head, tail, content } -// TestHashSignatureBelowThreshold verifies that a file below headTailMin -// is hashed in full and compared directly: head, tail, and content all -// carry the whole-file SHA-256, with no separate end-window step. -func TestHashSignatureBelowThreshold(t *testing.T) { +// TestHashSignature checks the three hashes of a file under +// wholeFileMax: head and tail are the SHA-256 of its first and last +// headTailWindow bytes, and content that of the whole file. In a file +// no larger than headTailWindow each window is the whole file, so all +// three are the whole-file hash. +func TestHashSignature(t *testing.T) { t.Parallel() dir := t.TempDir() @@ -87,8 +107,7 @@ func TestHashSignatureBelowThreshold(t *testing.T) { }{ {"one-byte", []byte("x")}, {"one-window", pattern(1, headTailWindow)}, - {"several-windows", pattern(2, 3*headTailWindow)}, - {"near-threshold", pattern(3, headTailMin-1)}, + {"several-windows", pattern(2, 3*headTailWindow+1)}, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { @@ -98,31 +117,31 @@ func TestHashSignatureBelowThreshold(t *testing.T) { head, tail, content := sig(t, p, int64(len(c.data))) - whole := hexSum(c.data) - if head != whole || tail != whole || content != whole { - t.Errorf("head=%s tail=%s content=%s, want all whole-file %s", - head, tail, content, whole) + window := min(len(c.data), headTailWindow) + wantHead := sha256.Sum256(c.data[:window]) + wantTail := sha256.Sum256(c.data[len(c.data)-window:]) + wantContent := sha256.Sum256(c.data) + + if head != wantHead || tail != wantTail || content != wantContent { + t.Errorf("head=%x tail=%x content=%x, want %x %x %x", + head, tail, content, wantHead, wantTail, wantContent) } }) } } -// TestHashSignatureEnds exercises the head and tail rungs, which apply -// only to files at least headTailMin. Sparse files keep the fixtures -// cheap: a difference in the first window changes only head, a -// difference in the last window changes only tail, and a difference -// between the windows changes neither end hash but does change the -// whole-file content rung (the file is below wholeFileMax). -// hashSignature leaves the content hash of a file this size to the -// content phase, so that rung is checked through a scan. +// TestHashSignatureEnds checks which hashes a one-byte difference +// changes in a file under wholeFileMax. Sparse files keep the fixtures +// cheap: a difference in the first window changes head but not tail, +// one in the last window changes tail but not head, and one between the +// windows changes neither but does change the whole-file content hash. func TestHashSignatureEnds(t *testing.T) { t.Parallel() dir := t.TempDir() - // Between headTailMin and wholeFileMax: the end-window gate is active - // and the content rung is a whole-file hash. - const size = int64(headTailMin + 2*1024*1024) + // Room for a byte between the two windows. + const size = int64(4 * headTailWindow) base := sparseFile(t, dir, "ends-base", size) headDiff := sparseFile(t, dir, "ends-head", size) @@ -134,9 +153,6 @@ func TestHashSignatureEnds(t *testing.T) { pokeAt(t, midDiff, size/2, []byte{1}) bHead, bTail, bContent := sig(t, base, size) - if bContent != "" { - t.Errorf("content = %q, want none from the hash phase", bContent) - } h, tl, _ := sig(t, headDiff, size) if h == bHead { @@ -156,16 +172,14 @@ func TestHashSignatureEnds(t *testing.T) { t.Error("a byte in the last window changed head") } - h, tl, _ = sig(t, midDiff, size) + h, tl, c := sig(t, midDiff, size) if h != bHead || tl != bTail { t.Error("a byte between the windows changed an end hash") } - // base and midDiff match on size, head, and tail, so the scan reads - // both for their content hashes. - c := scanContents(t, dir, base, midDiff) - if c[midDiff] == c[base] { - t.Error("whole-file content rung ignored a byte between the windows") + if c == bContent { + t.Error("the whole-file content hash ignored a byte between the " + + "windows") } } @@ -174,22 +188,25 @@ func TestHashSignatureErrors(t *testing.T) { dir := t.TempDir() - // A missing file: an error, and every hash left empty. + var zero [sha256.Size]byte + + // A missing file: an error, and every hash left zero. head, tail, content, err := hashSignature(filepath.Join(dir, "missing"), 1) if err == nil { t.Error("no error for a missing file") } - if head != "" || tail != "" || content != "" { - t.Errorf("missing file returned hashes: %q %q %q", head, tail, content) + if head != zero || tail != zero || content != zero { + t.Errorf("missing file returned hashes: %x %x %x", head, tail, content) } // A zero-length file has constant hashes and is never opened: even // a missing path succeeds. + empty := sha256.Sum256(nil) + head, tail, content, err = hashSignature(filepath.Join(dir, "missing"), 0) - if err != nil || - head != emptyHash || tail != emptyHash || content != emptyHash { - t.Errorf("empty: head=%q tail=%q content=%q err=%v, "+ + if err != nil || head != empty || tail != empty || content != empty { + t.Errorf("empty: head=%x tail=%x content=%x err=%v, "+ "want constant hashes", head, tail, content, err) } @@ -202,8 +219,8 @@ func TestHashSignatureErrors(t *testing.T) { t.Error("no error when the stat size exceeds the file size") } - if head != "" || tail != "" || content != "" { - t.Errorf("shrunk file returned hashes: %q %q %q", head, tail, content) + if head != zero || tail != zero || content != zero { + t.Errorf("shrunk file returned hashes: %x %x %x", head, tail, content) } } @@ -253,79 +270,46 @@ func pokeAt(t *testing.T, path string, off int64, data []byte) { } } -// scanContents scans dir into a fresh database and returns the content -// hash recorded for each file, by path, failing the test if one of want -// has none. A file of headTailMin or more gets a content hash only when -// it is scanned with a file of the same size, head, and tail. -func scanContents(t *testing.T, dir string, - want ...string, -) map[string]string { - t.Helper() - - db := openTestDB(t) - syncTree(t, db, dir) - - contents := make(map[string]string) - for _, r := range dbRecords(t, db) { - contents[r.path] = r.content - } - - for _, p := range want { - if contents[p] == "" { - t.Fatalf("%s: no content hash", p) - } - } - - return contents -} - -// TestContentRungBoundary checks the 50 MiB boundary between the two -// content rungs: just below it the whole file is hashed and any byte -// difference shows; at the boundary only the gigabyte-spaced samples are -// hashed, so a difference outside a sample window is invisible. The -// files of each pair match on size, head, and tail, so the scan reads -// both for their content hashes. -func TestContentRungBoundary(t *testing.T) { +// TestContentHashBoundary checks the 100 MiB boundary between the two +// content hashes through a scan and report. The files of each pair +// differ in one byte that neither end window nor any sample covers. +// Just below the boundary the whole file is hashed, so that pair is not +// reported. At the boundary only the samples are hashed, so that pair +// is reported as duplicates. +func TestContentHashBoundary(t *testing.T) { t.Parallel() dir := t.TempDir() + db := openTestDB(t) // A byte that lands outside the single [0, sampleWindow) sample a - // sub-gigabyte file has, but well inside the file. + // sub-gigabyte file has, and outside both end windows. const off = 10 * 1024 * 1024 - // Just under the boundary: the whole-file rung sees the poked byte. - under := int64(wholeFileMax - 1) - underBase := sparseFile(t, dir, "under-base", under) - underPoked := sparseFile(t, dir, "under-poked", under) + under := sparseFile(t, dir, "under-a", wholeFileMax-1) + underPoked := sparseFile(t, dir, "under-b", wholeFileMax-1) + at := sparseFile(t, dir, "at-a", wholeFileMax) + atPoked := sparseFile(t, dir, "at-b", wholeFileMax) pokeAt(t, underPoked, off, []byte{1}) - - // At the boundary: only [0, sampleWindow) is sampled, so the poked - // byte at off is invisible and the two content hashes match. - at := int64(wholeFileMax) - atBase := sparseFile(t, dir, "at-base", at) - atPoked := sparseFile(t, dir, "at-poked", at) - pokeAt(t, atPoked, off, []byte{1}) - c := scanContents(t, dir, underBase, underPoked, atBase, atPoked) + syncTree(t, db, dir) - if c[underBase] == c[underPoked] { - t.Error("whole-file rung ignored a byte difference below wholeFileMax") - } + want := []string{at, atPoked} - if c[atBase] != c[atPoked] { - t.Error("sampled rung saw a byte outside every sample window") + groups := dupeGroups(t, db) + if len(groups) != 1 || !slices.Equal(groups[0].paths, want) { + t.Fatalf("groups = %+v, want only the pair %q, not %q", groups, want, + []string{under, underPoked}) } } -// TestContentRungMultiGigabyte exercises the sampled rung across several -// gigabytes using sparse files: a difference inside the third sample -// window (at offset 2*sampleStride) changes the hash, while a difference -// in the gap after it does not. The three files match on size, head, -// and tail, so the scan reads each for its content hash. -func TestContentRungMultiGigabyte(t *testing.T) { +// TestContentHashMultiGigabyte exercises the sampled content hash +// across several gigabytes using sparse files: a difference inside the +// third sample window (at offset 2*sampleStride) changes the hash, +// while a difference in the gap after it does not. +func TestContentHashMultiGigabyte(t *testing.T) { t.Parallel() dir := t.TempDir() @@ -343,433 +327,19 @@ func TestContentRungMultiGigabyte(t *testing.T) { pokeAt(t, inSample, thirdSample, []byte{1}) pokeAt(t, inGap, gap, []byte{1}) - c := scanContents(t, dir, base, inSample, inGap) + _, _, cBase := sig(t, base, size) + _, _, cInSample := sig(t, inSample, size) + _, _, cInGap := sig(t, inGap, size) - if c[inSample] == c[base] { + if cInSample == cBase { t.Error("sample at 2 GiB was not read: difference there was invisible") } - if c[inGap] != c[base] { + if cInGap != cBase { t.Error("a byte in an unsampled gap changed the content hash") } } -// sparseFileWithoutMatch writes name in dir as a sparse file of size -// bytes, next to another file of that size whose first byte differs. A -// scan then reads the file's head and tail, since its size is shared, -// but finds no file matching them, so it gets no content hash. -func sparseFileWithoutMatch(t *testing.T, dir, name string, - size int64, -) string { - t.Helper() - - p := sparseFile(t, dir, name, size) - other := sparseFile(t, dir, name+"-other-head", size) - - pokeAt(t, other, 0, []byte{1}) - - return p -} - -// TestScanContentGate checks that a file of headTailMin or more is read -// for its content hash only when its size, head, and tail match another -// file's: a same-size pair whose heads differ and one whose tails differ -// get no content hash and are not reported, while an identical pair is -// read and reported. -func TestScanContentGate(t *testing.T) { - t.Parallel() - - dir := t.TempDir() - db := openTestDB(t) - - // Three sizes, so that no pair meets another. - headA := sparseFile(t, dir, "head-a", headTailMin) - headB := sparseFile(t, dir, "head-b", headTailMin) - tailA := sparseFile(t, dir, "tail-a", headTailMin+1) - tailB := sparseFile(t, dir, "tail-b", headTailMin+1) - same := []string{ - sparseFile(t, dir, "same-a", headTailMin+2), - sparseFile(t, dir, "same-b", headTailMin+2), - } - - pokeAt(t, headB, 0, []byte{1}) - pokeAt(t, tailB, headTailMin, []byte{1}) // its last byte - - syncTree(t, db, dir) - - recs := dbRecords(t, db) - for _, p := range []string{headA, headB, tailA, tailB} { - r := recordByPath(t, recs, p) - if r.head == "" || r.tail == "" || r.content != "" { - t.Errorf("%s: head = %q tail = %q content = %q, "+ - "want head and tail only", p, r.head, r.tail, r.content) - } - } - - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, same) { - t.Fatalf("groups = %+v, want only the identical pair %q", - groups, same) - } -} - -// TestScanContentAcrossOperands checks that a stored file gets its -// content hash when a later scan of a separate operand brings its -// match: tree A's file has a head and tail but no content hash until -// tree B, holding an identical file, is scanned. -func TestScanContentAcrossOperands(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - a := sparseFileWithoutMatch(t, t.TempDir(), "a", headTailMin) - - syncTree(t, db, filepath.Dir(a)) - - if r := recordByPath(t, dbRecords(t, db), a); r.head == "" || r.content != "" { - t.Fatalf("after scanning A: %+v, want head and tail only", r) - } - - b := sparseFile(t, t.TempDir(), "b", headTailMin) - - syncTree(t, db, filepath.Dir(b)) - - recs := dbRecords(t, db) - if r := recordByPath(t, recs, a); r.content == "" { - t.Fatalf("after scanning B: %+v, want A's file content-hashed", r) - } - - want := []string{a, b} - slices.Sort(want) - - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, want) { - t.Fatalf("groups = %+v, want the pair %q", groups, want) - } -} - -// TestScanContentWithinOperand checks that a rescan adding a match next -// to an unchanged stored file gives the stored file its content hash, -// though the hash phase leaves it alone as unchanged. -func TestScanContentWithinOperand(t *testing.T) { - t.Parallel() - - dir := t.TempDir() - db := openTestDB(t) - stored := sparseFileWithoutMatch(t, dir, "d1", headTailMin) - - syncTree(t, db, dir) - - added := sparseFile(t, dir, "d2", headTailMin) - - st := syncTree(t, db, dir) - if st != (scanStats{walked: 3, added: 1, unchanged: 2}) { - t.Fatalf("rescan stats = %+v, want 1 added 2 unchanged", st) - } - - want := []string{stored, added} - - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, want) { - t.Fatalf("groups = %+v, want the pair %q", groups, want) - } -} - -// TestScanContentStalePartners checks that a stored file outside the -// operand that has vanished, or changed, since it was recorded is not -// read, and that its match inside the operand is not read either: the -// match has no other partner left, so neither gets a content hash and -// no duplicate is reported. -func TestScanContentStalePartners(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - dirA := t.TempDir() - gone := sparseFileWithoutMatch(t, dirA, "gone", headTailMin) - changed := sparseFileWithoutMatch(t, dirA, "changed", headTailMin+1) - - syncTree(t, db, dirA) - - before := dbRecords(t, db) - - err := os.Remove(gone) - if err != nil { - t.Fatal(err) - } - - future := time.Now().Add(time.Hour) - - err = os.Chtimes(changed, future, future) - if err != nil { - t.Fatal(err) - } - - dirB := t.TempDir() - sparseFile(t, dirB, "gone-copy", headTailMin) - sparseFile(t, dirB, "changed-copy", headTailMin+1) - - st := syncTree(t, db, dirB) - if st != (scanStats{walked: 2, added: 2}) { - t.Errorf("stats = %+v, want 2 added and nothing skipped", st) - } - - recs := dbRecords(t, db) - for _, r := range recs { - if r.content != "" { - t.Errorf("%s: content = %q, want none: its only match is stale", - r.path, r.content) - } - } - - for _, old := range before { - if r := recordByPath(t, recs, old.path); r != old { - t.Errorf("record = %+v, want it left as %+v", r, old) - } - } - - if groups := dupeGroups(t, db); len(groups) != 0 { - t.Errorf("groups = %+v, want none", groups) - } -} - -// TestScanContentHashedStalePartners checks that stored matches outside -// the operand that already have a content hash are checked like any -// other: once one has vanished and the other has changed, a copy of -// them scanned in another tree has no match left, so it is not read and -// is not reported as their duplicate. -func TestScanContentHashedStalePartners(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - dirA := t.TempDir() - stored := []string{ - sparseFile(t, dirA, "changed", headTailMin), - sparseFile(t, dirA, "gone", headTailMin), - } - - // The two stored files match, so this scan gives both a content - // hash. - syncTree(t, db, dirA) - - err := os.Remove(stored[1]) - if err != nil { - t.Fatal(err) - } - - future := time.Now().Add(time.Hour) - - err = os.Chtimes(stored[0], future, future) - if err != nil { - t.Fatal(err) - } - - b := sparseFile(t, t.TempDir(), "copy", headTailMin) - - st := syncTree(t, db, filepath.Dir(b)) - if st != (scanStats{walked: 1, added: 1}) { - t.Errorf("stats = %+v, want 1 added and nothing skipped", st) - } - - recs := dbRecords(t, db) - if r := recordByPath(t, recs, b); r.content != "" { - t.Errorf("copy: content = %q, want none: its only matches are stale", - r.content) - } - - // The stored records lie outside the operand and are left as they - // are, so they still group with each other, but not with the copy. - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, stored) { - t.Errorf("groups = %+v, want only the stored pair %q", groups, stored) - } -} - -// TestScanContentSameSecondRewrite is TestScanContentStalePartners for -// a stored file rewritten in place at the same size with an mtime later -// in the same second than recorded: the file counts as changed, so -// neither it nor its match inside the operand is read. -func TestScanContentSameSecondRewrite(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - dirA := t.TempDir() - changed := sparseFileWithoutMatch(t, dirA, "changed", headTailMin) - - first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC) - - err := os.Chtimes(changed, first, first) - if err != nil { - t.Fatal(err) - } - - syncTree(t, db, dirA) - - before := dbRecords(t, db) - - // Rewrite one byte in place, keeping the size. - pokeAt(t, changed, headTailMin/2, []byte{1}) - - later := first.Add(500 * time.Millisecond) - - err = os.Chtimes(changed, later, later) - if err != nil { - t.Fatal(err) - } - - dirB := t.TempDir() - sparseFile(t, dirB, "changed-copy", headTailMin) - - st := syncTree(t, db, dirB) - if st != (scanStats{walked: 1, added: 1}) { - t.Errorf("stats = %+v, want 1 added and nothing skipped", st) - } - - recs := dbRecords(t, db) - for _, r := range recs { - if r.content != "" { - t.Errorf("%s: content = %q, want none: its only match is stale", - r.path, r.content) - } - } - - for _, old := range before { - if r := recordByPath(t, recs, old.path); r != old { - t.Errorf("record = %+v, want it left as %+v", r, old) - } - } - - if groups := dupeGroups(t, db); len(groups) != 0 { - t.Errorf("groups = %+v, want none", groups) - } -} - -// TestScanContentReadFailure checks that a failed content read is -// counted as skipped and leaves the record without a content hash, and -// that a later scan tries the read again. -func TestScanContentReadFailure(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - a := sparseFileWithoutMatch(t, t.TempDir(), "a", headTailMin) - - syncTree(t, db, filepath.Dir(a)) - - // lstat still works on the unreadable file, so it passes the check - // and fails only when it is read. - err := os.Chmod(a, 0) - if err != nil { - t.Fatal(err) - } - - dirB := t.TempDir() - b := sparseFile(t, dirB, "b", headTailMin) - - st := syncTree(t, db, dirB) - if st != (scanStats{walked: 1, added: 1, skipped: 1}) { - t.Fatalf("stats = %+v, want 1 added 1 skipped", st) - } - - if r := recordByPath(t, dbRecords(t, db), a); r.content != "" { - t.Fatalf("unreadable file: %+v, want no content hash", r) - } - - err = os.Chmod(a, 0o600) - if err != nil { - t.Fatal(err) - } - - st = syncTree(t, db, dirB) - if st != (scanStats{walked: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 unchanged", st) - } - - want := []string{a, b} - slices.Sort(want) - - groups := dupeGroups(t, db) - if len(groups) != 1 || !slices.Equal(groups[0].paths, want) { - t.Fatalf("groups = %+v, want the pair %q after the retry", - groups, want) - } -} - -// TestScanContentCheckError checks that a stored file the content phase -// cannot lstat, for a reason other than its being gone, is counted as -// skipped and does not count as a match. -func TestScanContentCheckError(t *testing.T) { - t.Parallel() - - db := openTestDB(t) - sub := filepath.Join(t.TempDir(), "sub") - - err := os.Mkdir(sub, 0o700) - if err != nil { - t.Fatal(err) - } - - sparseFileWithoutMatch(t, sub, "a", headTailMin) - syncTree(t, db, sub) - - // Without search permission on its directory, the stored file's - // lstat fails with permission denied. - err = os.Chmod(sub, 0) - if err != nil { - t.Fatal(err) - } - - t.Cleanup(func() { - //nolint:gosec // removing the directory needs its search bit back - _ = os.Chmod(sub, 0o700) - }) - - b := sparseFile(t, t.TempDir(), "b", headTailMin) - - st := syncTree(t, db, filepath.Dir(b)) - if st != (scanStats{walked: 1, added: 1, skipped: 1}) { - t.Fatalf("stats = %+v, want 1 added 1 skipped", st) - } - - if r := recordByPath(t, dbRecords(t, db), b); r.content != "" { - t.Errorf("b: content = %q, want none: its only match could not be "+ - "checked", r.content) - } -} - -// TestScanContentHardlinks checks that the content phase stores the -// content hash of a hard-linked file on every one of its links. -func TestScanContentHardlinks(t *testing.T) { - t.Parallel() - - dir := t.TempDir() - db := openTestDB(t) - a := sparseFile(t, dir, "a", headTailMin) - b := filepath.Join(dir, "b") - - err := os.Link(a, b) - if err != nil { - t.Fatal(err) - } - - c := sparseFile(t, dir, "copy", headTailMin) - - st := syncTree(t, db, dir) - if st != (scanStats{walked: 3, added: 3}) { - t.Fatalf("stats = %+v, want 3 added", st) - } - - recs := dbRecords(t, db) - - want := recordByPath(t, recs, c).content - if want == "" { - t.Fatal("the copy has no content hash") - } - - for _, p := range []string{a, b} { - if got := recordByPath(t, recs, p).content; got != want { - t.Errorf("%s: content = %q, want %q", p, got, want) - } - } -} - // collectWalk runs a walk over roots and returns the emitted records // and the number of warning events. func collectWalk(t *testing.T, roots []string, oneFS bool, @@ -1137,12 +707,114 @@ func TestSubdirJobStatError(t *testing.T) { } } +// TestWalkPassesOverRemovedPaths removes a file and a directory after +// their parent was read, as when they are deleted while a scan runs. +// The walk then reaches them, the directory also as one to read and as +// an operand. None of that is an error: the walk sends no event. +func TestWalkPassesOverRemovedPaths(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + file := writeFile(t, dir, "file", []byte("x")) + writeFile(t, dir, "sub/f", []byte("y")) + sub := filepath.Join(dir, "sub") + fileEntry := dirEntryFor(t, dir, "file") + subEntry := dirEntryFor(t, dir, "sub") + + for _, p := range []string{file, sub} { + err := os.RemoveAll(p) + if err != nil { + t.Fatal(err) + } + } + + ctx := t.Context() + events := make(chan walkEvent, 4) + + emitFile(ctx, file, fileEntry, walkRules{}, events) + + _, descend := subdirJob(ctx, sub, subEntry, dirJob{path: dir}, false, + walkRules{}, events) + subs := walkOneDir(ctx, dirJob{path: sub}, false, walkRules{}, events) + jobs := seedRoot(ctx, sub, walkRules{}, events) + + close(events) + + for ev := range events { + t.Errorf("event %+v for a removed path, want none", ev) + } + + if descend || len(subs) != 0 || len(jobs) != 0 { + t.Errorf("descend = %t, subdirectories %+v, jobs %+v, want none", + descend, subs, jobs) + } +} + +// TestScanWarnsAboutUnreadablePaths checks that a permission error is +// still warned about and counted as skipped, unlike a removed path. The +// walk reaches an unreadable directory, and a file in a directory that +// can be read but not searched. Another such file is an operand. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestScanWarnsAboutUnreadablePaths(t *testing.T) { + if os.Geteuid() == 0 { + t.Skip("root ignores file permissions") + } + + stderr := captureStderr(t) + dir := t.TempDir() + tree := filepath.Join(dir, "tree") + walked := writeFile(t, tree, "unsearchable/f", []byte("x")) + operand := writeFile(t, dir, "unsearchable/f", []byte("y")) + unreadable := filepath.Join(tree, "unreadable") + + err := os.Mkdir(unreadable, 0o750) + if err != nil { + t.Fatal(err) + } + + modes := map[string]fs.FileMode{ + unreadable: 0, + filepath.Dir(walked): 0o400, + filepath.Dir(operand): 0o400, + } + + for p, mode := range modes { + err = os.Chmod(p, mode) + if err != nil { + t.Fatal(err) + } + + // Runs before t.TempDir's own cleanup, which must delete the files. + t.Cleanup(func() { + //nolint:gosec // removing the files needs the search bit back + _ = os.Chmod(p, 0o700) + }) + } + + st := syncTree(t, openTestDB(t), tree, operand) + if st != (scanStats{skipped: 3}) { + t.Errorf("stats = %+v, want 3 skipped and nothing else", st) + } + + got := stderr() + for _, warning := range []string{ + "walk " + unreadable + ": open " + unreadable + ": permission denied\n", + "stat " + walked + ": lstat " + walked + ": permission denied\n", + "walk " + operand + ": lstat " + operand + ": permission denied\n", + } { + if !strings.Contains(got, warning) { + t.Errorf("stderr = %q, want %q", got, warning) + } + } +} + // syncOneFilesystem scans roots into db with -x and returns the scan // stats. Every device pruneRoots looks up, and the device each // operand's walk compares its subdirectories with, comes from devices // instead of from lstat, so a test can put a path on a filesystem that -// no directory here is on. Unlike runPhases, it skips walkableRoots, -// the check for a cancelled walk, and the content phase. +// no directory here is on. Unlike runPhases, it skips walkableRoots and +// the check for a cancelled walk. func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string, devices map[string]uint64, ) scanStats { @@ -1179,8 +851,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string, dispatchDirs(ctx, initial, jobs, subdirs) - changed, unhashed := s.walkPhase(events, roots) - s.partition(changed, unhashed) + s.walkPhase(events, roots) err = s.hashPhase(ctx, 2) if err != nil { @@ -1198,11 +869,10 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string, // TestScanOneFilesystemKeepsOtherFilesystemRecords checks that with -x a // record below a directory the walk does not enter, because it is on // another filesystem than its operand, counts as outside the operands: -// it is not deleted, and its size joins the size census. The record of -// a gone file that is not below that directory is still deleted, even -// when its path begins with the directory's name. The operand is given -// a device that no directory here is on, so no second real filesystem -// is needed. +// it is not deleted. The record of a gone file that is not below that +// directory is still deleted, even when its path begins with the +// directory's name. The operand is given a device that no directory +// here is on, so no second real filesystem is needed. func TestScanOneFilesystemKeepsOtherFilesystemRecords(t *testing.T) { t.Parallel() @@ -1220,10 +890,6 @@ func TestScanOneFilesystemKeepsOtherFilesystemRecords(t *testing.T) { t.Fatal(err) } - // The same size as mnt/f, so it is hashed only if mnt/f's size is - // in the census. - added := writeFile(t, dir, "added", pattern(2, 100)) - info, err := os.Lstat(dir) if err != nil { t.Fatal(err) @@ -1239,11 +905,6 @@ func TestScanOneFilesystemKeepsOtherFilesystemRecords(t *testing.T) { recs := dbRecords(t, db) recordByPath(t, recs, mounted) - if recordByPath(t, recs, added).head == "" { - t.Errorf("%s not hashed: the census lacks the size of %s", - added, mounted) - } - if slices.Contains(recordPaths(recs), gone) { t.Errorf("record for %s kept: the file is gone and not below mnt", gone) @@ -1427,12 +1088,13 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) { } } -// buildSmokeTree recreates the README smoke-test filesystem layout -// with deterministic content and returns the tree root. +// buildSmokeTree recreates the tree of the test under README +// "Definition of done" with deterministic content and returns the tree +// root. func buildSmokeTree(t *testing.T) string { t.Helper() - dir := t.TempDir() + dir := realTempDir(t) one := pattern(10, 2000) f1 := pattern(30, 3000) f2 := pattern(40, 100) @@ -1591,10 +1253,9 @@ func TestScanPipeline(t *testing.T) { // TestScanProgressNamesEachPhase checks the plain progress lines of a // first scan, and of a rescan of one operand holding two of the // database's records. Each phase's first line names it with its place -// among the five: load counts the records before reading them, the -// content phase searches the database, and the walk estimates its total -// from the records under its operand, or says its ETA is unknown when -// there are none. +// among the four: load gets the number of records before reading them, +// and the walk estimates its total from the records under its operand, +// or says its time remaining is unknown when there are none. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestScanProgressNamesEachPhase(t *testing.T) { @@ -1609,16 +1270,15 @@ func TestScanProgressNamesEachPhase(t *testing.T) { syncTree(t, db, filepath.Join(dir, "a")) rescan := stderr()[len(first):] - load := "[1/5] load: counting the records, elapsed 0s, eta unknown" - content := "[5/5] content: searching the database, elapsed 0s, eta unknown" + load := "[1/4] load: counting the records, elapsed 0s, remain unknown" cases := map[string][]string{ first: { - load, "[2/5] walk: 0 files, 0 files/s, elapsed 0s, eta unknown", - "[3/5] hash: [0/", "[4/5] update: [0/", content, + load, "[2/4] walk: 0 files, 0 files/s, elapsed 0s, remain unknown", + "[3/4] hash: [0/", "[4/4] update: [0/", }, rescan: { - load, "[2/5] walk: [0/~2] 0%, 0 files/s, elapsed 0s, eta unknown", - "[3/5] hash: [0/", "[4/5] update: [0/", content, + load, "[2/4] walk: [0/~2] 0%, 0 files/s, elapsed 0s, remain unknown", + "[3/4] hash: [0/", "[4/4] update: [0/", }, } @@ -1644,7 +1304,7 @@ func TestScanProgressNamesEachPhase(t *testing.T) { } // The load phase's total is the count of the records. - loaded := fmt.Sprintf("\n[1/5] load: [%d/%d] 100%%, ", + loaded := fmt.Sprintf("\n[1/4] load: [%d/%d] 100%%, ", smokeTreeFiles, smokeTreeFiles) if !strings.Contains(rescan, loaded) { t.Errorf("rescan stderr = %q, want %q", rescan, loaded) @@ -1657,8 +1317,7 @@ func TestSyncScanUnchangedReuse(t *testing.T) { dir := t.TempDir() db := openTestDB(t) a := writeFile(t, dir, "a.bin", pattern(1, 500)) - - writeFile(t, dir, "b.bin", pattern(2, 600)) + b := writeFile(t, dir, "b.bin", pattern(2, 600)) st := syncTree(t, db, dir) if st != (scanStats{walked: 2, added: 2}) { @@ -1666,22 +1325,33 @@ func TestSyncScanUnchangedReuse(t *testing.T) { } // An immediate rescan reuses every record without reading file - // contents. Prove the files are not re-read by corrupting a stored - // hash and observing that it survives the rescan. + // contents. Prove it: make every file unreadable, so that a read + // would fail and count as skipped, and corrupt a stored hash, which + // must survive the rescan. + for _, p := range []string{a, b} { + err := os.Chmod(p, 0) + if err != nil { + t.Fatal(err) + } + } + + sentinel := sha256.Sum256([]byte("sentinel")) + _, err := db.ExecContext(context.Background(), - "UPDATE files SET head = 'sentinel' WHERE path = ?", []byte(a)) + "UPDATE files SET head = ? WHERE path = ?", sentinel[:], []byte(a)) if err != nil { t.Fatal(err) } st = syncTree(t, db, dir) if st != (scanStats{walked: 2, unchanged: 2}) { - t.Fatalf("rescan stats = %+v, want 2 unchanged", st) + t.Fatalf("rescan stats = %+v, want 2 unchanged and none skipped", + st) } - if r := recordByPath(t, dbRecords(t, db), a); r.head != "sentinel" { - t.Fatalf("head = %q, want sentinel (file must not be re-read)", - r.head) + if r := recordByPath(t, dbRecords(t, db), a); r.head != sentinel { + t.Fatalf("head = %x, want the sentinel %x (file must not be re-read)", + r.head, sentinel) } } @@ -1721,91 +1391,257 @@ func assertWholeFileHashed(t *testing.T, db *sql.DB, path string, t.Helper() r := recordByPath(t, dbRecords(t, db), path) - if want := hexSum(data); r.head != want || r.tail != want || + if want := sha256.Sum256(data); r.head != want || r.tail != want || r.content != want { - t.Fatalf("head, tail, content = %q, %q, %q, want %q for each", + t.Fatalf("head, tail, content = %x, %x, %x, want %x for each", r.head, r.tail, r.content, want) } } +// TestScanStoresRawHashes checks what the database holds for a file +// larger than both end windows, whose three hashes all differ: head, +// tail and content are each a BLOB of the 32 bytes of the SHA-256 the +// scan computed. +func TestScanStoresRawHashes(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + db := openTestDB(t) + data := pattern(3, 3*headTailWindow) + p := writeFile(t, dir, "f.bin", data) + + syncTree(t, db, dir) + + var ( + blobs bool + head, tail, content []byte + ) + + err := db.QueryRowContext(t.Context(), + "SELECT typeof(head) = 'blob' AND typeof(tail) = 'blob' AND "+ + "typeof(content) = 'blob', head, tail, content "+ + "FROM files WHERE path = ?", []byte(p)). + Scan(&blobs, &head, &tail, &content) + if err != nil { + t.Fatal(err) + } + + if !blobs { + t.Error("head, tail and content are not all BLOBs") + } + + wantHead := sha256.Sum256(data[:headTailWindow]) + wantTail := sha256.Sum256(data[len(data)-headTailWindow:]) + wantContent := sha256.Sum256(data) + + if !bytes.Equal(head, wantHead[:]) || !bytes.Equal(tail, wantTail[:]) || + !bytes.Equal(content, wantContent[:]) { + t.Errorf("head, tail, content = %x, %x, %x, want %x, %x, %x", + head, tail, content, wantHead, wantTail, wantContent) + } +} + +// TestScanStoresDiskSize scans a sparse file, which takes less space on +// disk than its length, once through its directory and once as the +// operand itself. Each scan stores the disk_size of the file's lstat. +func TestScanStoresDiskSize(t *testing.T) { + t.Parallel() + + const length = 1024 * 1024 + + p := sparseFile(t, t.TempDir(), "sparse", length) + pokeAt(t, p, 0, []byte("x")) + + unix.Sync() // see diskSizeOf + + size := diskSizeOf(t, p) + if size == 0 || size >= length { + t.Fatalf("the file takes %d bytes on disk, want more than none "+ + "and fewer than its length %d", size, length) + } + + for _, operand := range []string{filepath.Dir(p), p} { + db := openTestDB(t) + + // ZFS may count the blocks during the scan; see diskSizeOf. + before := diskSizeOf(t, p) + + syncTree(t, db, operand) + + after := diskSizeOf(t, p) + + got := recordByPath(t, dbRecords(t, db), p).diskSize + if got < before || got > after { + t.Errorf("scan of %s: disk_size = %d, want %d to %d", + operand, got, before, after) + } + } +} + +// TestSyncScanRewriteUpdatesDiskSize writes into the hole of a scanned +// sparse file and moves its mtime on. The rescan hashes the file again +// and rewrites its record with the new disk_size. +func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + db := openTestDB(t) + p := sparseFile(t, dir, "sparse", 1024*1024) + pokeAt(t, p, 0, []byte("x")) + + syncTree(t, db, dir) + + recorded := recordByPath(t, dbRecords(t, db), p).diskSize + + pokeAt(t, p, 512*1024, pattern(7, 64*1024)) + + // The write may leave the mtime the scan recorded; a later one makes + // the rescan read the file again. + later := time.Now().Add(time.Hour) + + err := os.Chtimes(p, later, later) + if err != nil { + t.Fatal(err) + } + + // Wait for ZFS to count the write; see diskSizeOf. + deadline := time.Now().Add(30 * time.Second) + for diskSizeOf(t, p) == recorded { + if time.Now().After(deadline) { + t.Fatalf("the file still takes %d bytes on disk 30 seconds "+ + "after the write into its hole", recorded) + } + + unix.Sync() + } + + // ZFS may still be counting, so the size may change during a rescan. + // Only a rescan with the same size on both sides of it can be compared + // exactly. A rescan that keeps the old size still fails, because the + // wait made before differ from it. + for { + before := diskSizeOf(t, p) + + st := syncTree(t, db, dir) + + after := diskSizeOf(t, p) + + if st != (scanStats{walked: 1, updated: 1}) { + t.Fatalf("rescan stats = %+v, want 1 updated", st) + } + + if before == after { + got := recordByPath(t, dbRecords(t, db), p).diskSize + if got != after { + t.Errorf("disk_size = %d, want %d", got, after) + } + + return + } + + if time.Now().After(deadline) { + t.Fatalf("the size on disk was still changing 30 seconds after "+ + "the write into its hole: %d, then %d", before, after) + } + + // A later mtime makes the next rescan read the file again. + later = later.Add(time.Second) + + err = os.Chtimes(p, later, later) + if err != nil { + t.Fatal(err) + } + } +} + +// TestSyncScanUnchangedUpdatesDiskSize gives a file's record a +// disk_size the file does not have, as when ZFS had not yet counted the +// file's blocks at the last scan. The rescan writes the file's own +// disk_size into the record without reading the file again. +func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + db := openTestDB(t) + p := writeFile(t, dir, "a.bin", pattern(1, 500)) + + syncTree(t, db, dir) + + // lstat counts 512-byte units, so no file takes 1 byte on disk. + _, err := db.ExecContext(t.Context(), + "UPDATE files SET disk_size = 1 WHERE path = ?", []byte(p)) + if err != nil { + t.Fatal(err) + } + + unix.Sync() // see diskSizeOf + + // ZFS may count the blocks during the rescan; see diskSizeOf. + before := diskSizeOf(t, p) + + // A file read again would count as updated. + st := syncTree(t, db, dir) + + after := diskSizeOf(t, p) + + if st != (scanStats{walked: 1, unchanged: 1}) { + t.Fatalf("rescan stats = %+v, want 1 unchanged", st) + } + + got := recordByPath(t, dbRecords(t, db), p).diskSize + if got < before || got > after { + t.Errorf("disk_size = %d, want %d to %d", got, before, after) + } +} + // TestSyncScanSameSecondRewrite rewrites a file in place at the same // size with an mtime later in the same second as the recorded one: the -// next scan must notice the change and re-hash the file. +// next scan must notice the change and re-hash the file. The operand is +// the file's directory, or the file itself, which scan stats without +// reading its directory. func TestSyncScanSameSecondRewrite(t *testing.T) { t.Parallel() - dir := t.TempDir() - db := openTestDB(t) - a := writeFile(t, dir, "a.bin", pattern(1, 500)) + for _, fileOperand := range []bool{false, true} { + t.Run(fmt.Sprintf("file operand %v", fileOperand), func(t *testing.T) { + t.Parallel() - // b.bin shares the size of a.bin, so a.bin is hashed. - writeFile(t, dir, "b.bin", pattern(2, 500)) + dir := t.TempDir() + db := openTestDB(t) + a := writeFile(t, dir, "a.bin", pattern(1, 500)) - first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC) + operand := dir + if fileOperand { + operand = a + } - err := os.Chtimes(a, first, first) - if err != nil { - t.Fatal(err) + first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC) + + err := os.Chtimes(a, first, first) + if err != nil { + t.Fatal(err) + } + + syncTree(t, db, operand) + + rewritten := pattern(3, 500) + writeFile(t, dir, "a.bin", rewritten) + + later := first.Add(500 * time.Millisecond) + + err = os.Chtimes(a, later, later) + if err != nil { + t.Fatal(err) + } + + st := syncTree(t, db, operand) + if st != (scanStats{walked: 1, updated: 1}) { + t.Fatalf("rescan stats = %+v, want 1 updated", st) + } + + assertWholeFileHashed(t, db, a, rewritten) + }) } - - syncTree(t, db, dir) - - rewritten := pattern(3, 500) - writeFile(t, dir, "a.bin", rewritten) - - later := first.Add(500 * time.Millisecond) - - err = os.Chtimes(a, later, later) - if err != nil { - t.Fatal(err) - } - - st := syncTree(t, db, dir) - if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st) - } - - assertWholeFileHashed(t, db, a, rewritten) -} - -// TestSyncScanOperandSameSecondRewrite is TestSyncScanSameSecondRewrite -// for files given to scan as operands, which scan stats without reading -// their directory. -func TestSyncScanOperandSameSecondRewrite(t *testing.T) { - t.Parallel() - - dir := t.TempDir() - db := openTestDB(t) - a := writeFile(t, dir, "a.bin", pattern(1, 500)) - - // b.bin shares the size of a.bin, so a.bin is hashed. - b := writeFile(t, dir, "b.bin", pattern(2, 500)) - - first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC) - - err := os.Chtimes(a, first, first) - if err != nil { - t.Fatal(err) - } - - syncTree(t, db, a, b) - - rewritten := pattern(3, 500) - writeFile(t, dir, "a.bin", rewritten) - - later := first.Add(500 * time.Millisecond) - - err = os.Chtimes(a, later, later) - if err != nil { - t.Fatal(err) - } - - st := syncTree(t, db, a, b) - if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st) - } - - assertWholeFileHashed(t, db, a, rewritten) } // TestSyncScanRewriteAfter2262 runs assertLateRewriteRehashed with an @@ -1836,9 +1672,6 @@ func assertLateRewriteRehashed(t *testing.T, late time.Time) { db := openTestDB(t) a := writeFile(t, dir, "a.bin", pattern(1, 500)) - // b.bin shares the size of a.bin, so a.bin is hashed. - writeFile(t, dir, "b.bin", pattern(2, 500)) - syncTree(t, db, dir) rewritten := pattern(3, 500) @@ -1867,8 +1700,8 @@ func assertLateRewriteRehashed(t *testing.T, late time.Time) { } st := syncTree(t, db, dir) - if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st) + if st != (scanStats{walked: 1, updated: 1}) { + t.Fatalf("rescan stats = %+v, want 1 updated", st) } assertWholeFileHashed(t, db, a, rewritten) @@ -2017,88 +1850,19 @@ func TestSyncScanOverlappingRoots(t *testing.T) { } } -func TestScanSkipsUniqueSizes(t *testing.T) { +// TestScanHashesUniqueSize checks that a file whose size no other file +// shares is hashed like every other file. +func TestScanHashesUniqueSize(t *testing.T) { t.Parallel() dir := t.TempDir() db := openTestDB(t) - a := writeFile(t, dir, "a.bin", pattern(1, 500)) + data := pattern(1, 500) + a := writeFile(t, dir, "a.bin", data) - writeFile(t, dir, "b.bin", pattern(2, 600)) + syncTree(t, db, dir) - // Neither size is shared, so neither file is read: both records - // are written without hashes and no duplicates are reported. - st := syncTree(t, db, dir) - if st != (scanStats{walked: 2, added: 2}) { - t.Fatalf("stats = %+v, want 2 added", st) - } - - recs := dbRecords(t, db) - for _, r := range recs { - if r.head != "" || r.tail != "" || r.content != "" { - t.Errorf("%s: head = %q tail = %q content = %q, want unhashed", - r.path, r.head, r.tail, r.content) - } - } - - if groups := dupeGroups(t, db); len(groups) != 0 { - t.Fatalf("groups = %+v, want none from unhashed records", groups) - } - - // A new same-size file makes 500 a shared size: the next scan - // hashes both the new file and the previously unhashed unchanged - // one, and they group as duplicates. - c := writeFile(t, dir, "c.bin", pattern(1, 500)) - - st = syncTree(t, db, dir) - if st != (scanStats{walked: 3, added: 1, updated: 1, unchanged: 1}) { - t.Fatalf("rescan stats = %+v, want 1 added 1 updated 1 unchanged", - st) - } - - groups := dupeGroups(t, db) - if len(groups) != 1 { - t.Fatalf("groups = %+v, want the a/c pair", groups) - } - - if want := []string{a, c}; !slices.Equal(groups[0].paths, want) { - t.Fatalf("group paths = %q, want %q", groups[0].paths, want) - } -} - -func TestTreesUnhashedNeverEqual(t *testing.T) { - t.Parallel() - - // Two trees identical except for same-name, same-size files without - // a content hash must not compare equal: their content is unknown. - // That holds for unhashed files (possible when the trees were - // scanned separately) and for files of headTailMin or more that - // have only a head and tail. - sum := hexSum(pattern(1, 100)) - shared := []scanRec{ - {path: "/x/t1/f1", size: 100, head: sum, tail: sum, content: sum}, - {path: "/x/t2/f1", size: 100, head: sum, tail: sum, content: sum}, - } - - cases := map[string][]scanRec{ - "unhashed": { - {path: "/x/t1/u", size: 50}, - {path: "/x/t2/u", size: 50}, - }, - "head and tail only": { - {path: "/x/t1/u", size: headTailMin, head: "h", tail: "t"}, - {path: "/x/t2/u", size: headTailMin, head: "h", tail: "t"}, - }, - } - - for name, unknown := range cases { - super, dirs := treeOf(t, append(slices.Clone(shared), unknown...)) - - if tg := collectTreeGroups(dirs, super, nil); len(tg) != 0 { - t.Errorf("%s: tree groups = %d, want 0 (the files may differ)", - name, len(tg)) - } - } + assertWholeFileHashed(t, db, a, data) } func TestScanHardlinksReadOnce(t *testing.T) { @@ -2125,7 +1889,8 @@ func TestScanHardlinksReadOnce(t *testing.T) { ra := recordByPath(t, recs, a) rb := recordByPath(t, recs, b) - if ra.head == "" || ra.head != rb.head || ra.tail != rb.tail { + if ra.head == [sha256.Size]byte{} || ra.head != rb.head || + ra.tail != rb.tail { t.Fatalf("hardlink hashes differ: %+v vs %+v", ra, rb) } @@ -2198,12 +1963,54 @@ func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) { if got := stderr(); strings.Count(got, warning) != 1 { t.Errorf("stderr = %q, want %q once", got, warning) } +} - // The content phase opens files the same way. - _, _, content, err := hashContentOnly(p, headTailMin) - if content != "" || !errors.Is(err, errNotRegular) { - t.Errorf("hashContentOnly on a FIFO = %q, %v; want no hash and %v", - content, err, errNotRegular) +// TestHashPhasePassesOverRemovedFiles removes two walked files before +// the hash phase reads them: one with no other path, and the first of +// two hard links to one file. Neither is an error. The phase prints +// nothing but its progress, counts nothing as skipped, and records only +// the hard link still there. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestHashPhasePassesOverRemovedFiles(t *testing.T) { + stderr := captureStderr(t) + dir := t.TempDir() + alone := writeFile(t, dir, "alone.bin", pattern(1, 300)) + a := writeFile(t, dir, "a.bin", pattern(2, 300)) + b := filepath.Join(dir, "b.bin") + + err := os.Link(a, b) + if err != nil { + t.Fatal(err) + } + + walked, _ := collectWalk(t, []string{dir}, false, 1) + + // a.bin sorts before b.bin, so the run of the two reads it first. + for _, p := range []string{alone, a} { + err = os.Remove(p) + if err != nil { + t.Fatal(err) + } + } + + s := &scanState{db: openTestDB(t), toHash: walked} + + err = s.hashPhase(t.Context(), 1) + if err != nil { + t.Fatalf("hashPhase = %v, want nil", err) + } + + if s.st.skipped != 0 || len(s.batch) != 1 || s.batch[0].path != b { + t.Errorf("skipped = %d, records %+v, want none skipped and a "+ + "record for %s alone", s.st.skipped, s.batch, b) + } + + for line := range strings.Lines(stderr()) { + if !strings.HasPrefix(line, scanPhase("hash")+": ") { + t.Errorf("stderr holds %q, want the hash phase's progress alone", + line) + } } } @@ -2503,33 +2310,24 @@ func TestUnderRoot(t *testing.T) { } // TestRunScanPassesOverDatabase scans a tree that holds the scan's own -// database, reached directly and through a symlinked directory. The -// walk passes over the database file, its -wal and -shm files and its -// .lock file with no line and no count, and the record an earlier scan -// left for the .lock file is deleted. A rescan of the unchanged tree -// updates nothing, and report lists none of the database's files. +// database, and in one case also a hard link to the database file. The +// walk knows the database's files by device ID and inode number, so it +// passes over the database file, its -wal and -shm files, its .lock +// file and the hard link with no line and no count. The record an +// earlier scan left for the .lock file is deleted. A rescan of the +// unchanged tree updates nothing, and report lists none of the +// database's files. func TestRunScanPassesOverDatabase(t *testing.T) { - for _, viaSymlink := range []bool{false, true} { - t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) { - base := t.TempDir() - tree := filepath.Join(base, "real", "tree") - operand := tree + for _, viaHardLink := range []bool{false, true} { + t.Run(fmt.Sprintf("via hard link %v", viaHardLink), func(t *testing.T) { + tree := realTempDir(t) + hardLink := filepath.Join(tree, "hardlink") writeFile(t, tree, "a", pattern(1, 100)) writeFile(t, tree, "b", pattern(1, 100)) writeFile(t, tree, "empty1", nil) writeFile(t, tree, "empty2", nil) - if viaSymlink { - err := os.Symlink(filepath.Join(base, "real"), - filepath.Join(base, "link")) - if err != nil { - t.Fatal(err) - } - - operand = filepath.Join(base, "link", "tree") - } - dbPath := filepath.Join(tree, "db.sqlite") t.Setenv(databaseEnv, dbPath) @@ -2539,7 +2337,7 @@ func TestRunScanPassesOverDatabase(t *testing.T) { } err = applyChanges(t.Context(), db, []scanRec{ - {path: filepath.Join(operand, "db.sqlite.lock")}, + {path: filepath.Join(tree, "db.sqlite.lock")}, }, nil, nil) if err != nil { t.Fatal(err) @@ -2547,16 +2345,29 @@ func TestRunScanPassesOverDatabase(t *testing.T) { closeScanDatabase(t.Context(), db, dbPath) - assertScanSummary(t, + if viaHardLink { + err = os.Link(dbPath, hardLink) + if err != nil { + t.Fatal(err) + } + } + + stderr := assertScanSummary(t, "4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)", - operand) - assertScanSummary(t, + tree) + stderr += assertScanSummary(t, "4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)", - operand) + tree) + + if strings.Contains(stderr, hardLink) { + t.Errorf("scan stderr = %q, want no line naming %s", + stderr, hardLink) + } report := runStdout(t, cmdReport) if strings.Contains(report, "db.sqlite") || - !strings.Contains(report, filepath.Join(operand, "empty2")) { + strings.Contains(report, hardLink) || + !strings.Contains(report, filepath.Join(tree, "empty2")) { t.Errorf("report = %q, want the empty files and none of "+ "the database's files", report) } @@ -2581,8 +2392,9 @@ func TestRunScanPassesOverDatabaseOperands(t *testing.T) { // assertScanSummary scans operands and fails the test unless the // summary line gives counts with nothing skipped, and the line naming -// the database is the only one that names one of its files. -func assertScanSummary(t *testing.T, counts string, operands ...string) { +// the database is the only one that names one of its files. It returns +// everything the scan printed to stderr. +func assertScanSummary(t *testing.T, counts string, operands ...string) string { t.Helper() stderr := scanOK(t, operands...) @@ -2596,4 +2408,6 @@ func assertScanSummary(t *testing.T, counts string, operands ...string) { t.Errorf("scan %q: stderr = %q, want only the line naming the "+ "database to name one of its files", operands, stderr) } + + return stderr } diff --git a/trees.go b/trees.go index 77275a5..554666f 100644 --- a/trees.go +++ b/trees.go @@ -22,6 +22,10 @@ type treeNode struct { digest [sha256.Size]byte fileCount int64 totalSize int64 + // diskSize is the bytes the directory's files take on disk. Two + // copies of one tree have the same totalSize, but their diskSize + // can differ. + diskSize int64 } // treesPhase names one phase of trees with its place among all of them, @@ -40,11 +44,11 @@ func treesPhase(name string) string { // database problem, including a missing database, is fatal. // // The work runs in four phases, each under a progress display of its -// own on stderr: open (open the database), read (count the records, -// then read them into the hierarchy), group (group the directories by -// digest) and write (write the groups to stdout). Each display ends -// before the next phase starts, and before an error is returned to be -// printed. +// own on stderr: open (open the database), read (read the number of +// records from the totals table, then read the records into the +// hierarchy), group (group the directories by digest) and write (write +// the groups to stdout). Each display ends before the next phase +// starts, and before an error is returned to be printed. func runTrees(ctx context.Context, stdout io.Writer) error { dbPath, source, err := databasePath() if err != nil { @@ -139,15 +143,16 @@ func readTree(ctx context.Context, db *sql.DB, // writeTreeGroups writes dupes to stdout as TSV, a header line and then // a row for every member of each group but its first, counting each -// group on prog once its rows are written. stdout may be the terminal -// prog is drawn on, so the lines go through prog's writer. It returns -// the number of rows (dupe trees) and the bytes they could reclaim. +// group on prog once its rows are written. The disk_size column is the +// dupe's own. stdout may be the terminal prog is drawn on, so the lines +// go through prog's writer. It returns the number of rows (dupe trees) +// and the bytes they could reclaim. func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode, prog *progress, ) (int, int64, error) { out := prog.writer(stdout) - _, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize") + _, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size") if err != nil { return 0, 0, fmt.Errorf("write stdout: %w", err) } @@ -159,9 +164,9 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode, for _, g := range dupes { first := g[0] for _, n := range g[1:] { - _, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n", + _, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\t%d\n", escapePath(first.path), escapePath(n.path), - first.fileCount, first.totalSize) + first.fileCount, first.totalSize, n.diskSize) if err != nil { return 0, 0, fmt.Errorf("write stdout: %w", err) } @@ -232,6 +237,7 @@ func (b *treeBuilder) add(r scanRec) { dir.entries = append(dir.entries, fileEntry(name, r)) dir.fileCount++ dir.totalSize += r.size + dir.diskSize += r.diskSize } // openDir starts the directory called name inside the innermost open @@ -270,6 +276,7 @@ func (b *treeBuilder) closeTo(n int) { "d\x00"+name+"\x00"+string(dir.digest[:])) dir.parent.fileCount += dir.fileCount dir.parent.totalSize += dir.totalSize + dir.parent.diskSize += dir.diskSize b.dirs = append(b.dirs, dir) } } @@ -286,18 +293,9 @@ func (b *treeBuilder) finish() (*treeNode, []*treeNode) { // name and its signature (size, head, tail, content); mtime is // excluded. func fileEntry(name string, r scanRec) string { - content := r.content - - // A record without a content hash has unknown content (README - // "Database"): give it a signature no other file can share, so - // trees containing it never compare equal. Real hashes are hex, so - // the NUL-prefixed form cannot collide. - if content == "" { - content = "unhashed\x00" + r.path - } - return "f\x00" + name + "\x00" + strconv.FormatInt(r.size, 10) + - "\x00" + r.head + "\x00" + r.tail + "\x00" + content + "\x00" + string(r.head[:]) + "\x00" + string(r.tail[:]) + + "\x00" + string(r.content[:]) } // collectTreeGroups groups directories by digest, counting each one on @@ -347,7 +345,8 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode, // digest is the SHA-256 of its child entries — files serialized with // name and signature, subdirectories with name and recursive digest — // sorted byte-lexicographically. Filenames cannot contain NUL or "/", -// so NUL delimiters are unambiguous. +// and every hash and digest is 32 bytes, so NUL delimiters are +// unambiguous. func (n *treeNode) computeDigest() { slices.Sort(n.entries) diff --git a/trees_test.go b/trees_test.go index 9249aa0..403762e 100644 --- a/trees_test.go +++ b/trees_test.go @@ -10,32 +10,31 @@ import ( "time" ) -// Signature hashes shared by the smoke-test records. -const ( - f1Head = "f1h" - f1Tail = "f1t" - f1Content = "f1c" - f2Head = "f2h" - f2Tail = "f2t" - f2Content = "f2c" -) - // treesOpenLine is the plain line of trees' open phase when it opens // the database at once; it prints it as the phase starts and as it ends. const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" -// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and -// /d/t2 are identical, /d/t3 differs from them only by one filename. +// smokeTreeRecs mirrors the tree of the test under README "Definition +// of done": /d/t1 and /d/t2 are identical, /d/t3 differs from them only +// by one filename. Each file takes one 4 KiB block on disk. func smokeTreeRecs() []scanRec { + f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c") + f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c") + return []scanRec{ - {size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"}, - {size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"}, - {size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t2/f1"}, - {size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t2/sub/f2"}, - {size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t3/f1"}, - {size: 100, head: f2Head, tail: f2Tail, content: f2Content, - path: "/d/t3/sub/f2renamed"}, + {size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail, + content: f1Content, path: "/d/t1/f1"}, + {size: 100, diskSize: 4096, head: f2Head, tail: f2Tail, + content: f2Content, path: "/d/t1/sub/f2"}, + {size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail, + content: f1Content, path: "/d/t2/f1"}, + {size: 100, diskSize: 4096, head: f2Head, tail: f2Tail, + content: f2Content, path: "/d/t2/sub/f2"}, + {size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail, + content: f1Content, path: "/d/t3/f1"}, + {size: 100, diskSize: 4096, head: f2Head, tail: f2Tail, + content: f2Content, path: "/d/t3/sub/f2renamed"}, } } @@ -153,13 +152,14 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) { for _, top := range []string{"/a", "/c"} { for _, p := range []string{"/b", "/b-x/f", "/b.txt", "/b/f"} { - content := "c" + content := testHash("c") if p == "/b-x/f" { - content = "other" + content = testHash("other") } recs = append(recs, scanRec{ - size: 1, head: "h", tail: "t", content: content, path: top + p, + size: 1, head: testHash("h"), tail: testHash("t"), + content: content, path: top + p, }) } } @@ -204,13 +204,33 @@ func TestRunTreesEscapesPaths(t *testing.T) { code, exitOK, stderr.String()) } - want := "first\tdupe\tfiles\tsize\n" + - `/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\n" + want := "first\tdupe\tfiles\tsize\tdisk_size\n" + + `/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n" if got := stdout.String(); got != want { t.Errorf("stdout = %q, want %q", got, want) } } +// TestRunTreesDiskSize runs trees on two copies of a tree whose files +// take different space on disk. The row's disk_size is the sum over the +// dupe copy's files, those in its subdirectory included. +func TestRunTreesDiskSize(t *testing.T) { + f, g := testHash("f"), testHash("g") + recs := []scanRec{ + {size: 10, diskSize: 4096, head: f, tail: f, content: f, path: "/a/f"}, + {size: 20, diskSize: 4096, head: g, tail: g, content: g, path: "/a/sub/g"}, + {size: 10, diskSize: 512, head: f, tail: f, content: f, path: "/b/f"}, + {size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"}, + } + + t.Setenv(databaseEnv, seedDatabase(t, recs)) + + want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n" + if got := runStdout(t, cmdTrees); got != want { + t.Errorf("stdout = %q, want %q", got, want) + } +} + // TestRunTreesProgressNamesEachPhase runs trees on the smoke-test // records with stderr not a terminal, and checks every line it prints // there. After the line naming the database, each phase prints a line @@ -232,7 +252,8 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) { code, exitOK, stderr()) } - report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n" + report := "first\tdupe\tfiles\tsize\tdisk_size\n" + + "/d/t1\t/d/t2\t2\t3100\t8192\n" if got := stdout.String(); got != report { t.Errorf("stdout = %q, want %q", got, report) } @@ -242,11 +263,11 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) { "trees: database " + path + " (from " + databaseEnv + ")", treesOpenLine, treesOpenLine, - "[2/4] read: counting the records, elapsed 0s, eta unknown", + "[2/4] read: counting the records, elapsed 0s, remain unknown", "[2/4] read: [6/6] 100%, ", - "[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown", + "[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown", "[3/4] group: [8/8] 100%, ", - "[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown", + "[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown", "[4/4] write: [1/1] 100%, ", "trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ", } @@ -297,7 +318,9 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) { shown = append(shown, line[strings.LastIndex(line, "\r")+1:]) } - want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"} + want := []string{ + "first\tdupe\tfiles\tsize\tdisk_size", "/d/t1\t/d/t2\t2\t3100\t8192", + } if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) { t.Errorf("terminal shows lines %q, want %q first", shown, want) } @@ -332,11 +355,10 @@ func TestTreeDigests(t *testing.T) { func TestTreeDigestContentSensitivity(t *testing.T) { t.Parallel() - const sharedTail = "same" - + same, c := testHash("same"), testHash("c") recs := []scanRec{ - {size: 10, head: sharedTail, tail: sharedTail, content: "c", path: "/r/a/f"}, - {size: 10, head: "DIFF", tail: sharedTail, content: "c", path: "/r/b/f"}, + {size: 10, head: same, tail: same, content: c, path: "/r/a/f"}, + {size: 10, head: testHash("DIFF"), tail: same, content: c, path: "/r/b/f"}, } _, dirs := treeOf(t, recs) @@ -397,9 +419,10 @@ func TestCollectTreeGroupsSiblings(t *testing.T) { // Identical sibling dirs share a parent, so their group cannot be // implied by a parent group and must be reported. + h, tl, c := testHash("h"), testHash("t"), testHash("c") recs := []scanRec{ - {size: 10, head: "h", tail: "t", content: "c", path: "/p/x1/f"}, - {size: 10, head: "h", tail: "t", content: "c", path: "/p/x2/f"}, + {size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"}, + {size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"}, } super, dirs := treeOf(t, recs) @@ -418,10 +441,11 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) { // /p/a and /q/b contain identical x subtrees, but /p/a has an // extra file, so the parents' digests differ and the x group must // be reported. + h, tl, c, e := testHash("h"), testHash("t"), testHash("c"), testHash("e") recs := []scanRec{ - {size: 10, head: "h", tail: "t", content: "c", path: "/p/a/x/f"}, - {size: 99, head: "e", tail: "e", content: "e", path: "/p/a/extra"}, - {size: 10, head: "h", tail: "t", content: "c", path: "/q/b/x/f"}, + {size: 10, head: h, tail: tl, content: c, path: "/p/a/x/f"}, + {size: 99, head: e, tail: e, content: e, path: "/p/a/extra"}, + {size: 10, head: h, tail: tl, content: c, path: "/q/b/x/f"}, } super, dirs := treeOf(t, recs)