diff --git a/TODO.md b/TODO.md index f2bafd6..88f517c 100644 --- a/TODO.md +++ b/TODO.md @@ -4,7 +4,7 @@ the tracker yet is filed as an issue first. - Branch from `next`. - Do the work, with tests, in small focused commits. -- Record the step at the top of Completed Steps, in the same commit as the work. +- Delete Next Step from `TODO.md`, in the same commit as the work. - Push the branch and open a PR against `next`. Its title ends with ` (closes #N)`. - An independent review gates each merge to `next`. Every finding is addressed @@ -15,8 +15,7 @@ - Pre-1.0. - The tracker is authoritative for the pre-1.0 backlog. The open issues under - the `1.0.0` milestone are what remains before the tag. This file records - history and process, not the queue. + the `1.0.0` milestone are what remains before the tag. # Next Step @@ -24,771 +23,6 @@ https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated here, because a copy drifts as soon as the tracker moves. -# Completed Steps - -Newest first. An entry ends with its date and its issue or branch, where those -are known. - -- `dwim` takes the `scan` flags `--workers`, `-x`, `--no-default-excludes` and - `--exclude`, with the same meaning. Both commands register and check them with - the same code (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/162) - -- The help of `sfdupes` and of `scan` names the database, its default path and - `SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the - order of the rows and the escapes. The help of `info` lists its lines. The - help of `dwim` says that it scans `/`, should run as root and keeps the scan - in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177) -- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither - enters nor records an excluded path, and keeps the records below it. An - operand at or below one is a fatal error (2026-11-11, - https://git.eeqj.de/sneak/sfdupes/issues/163) - -- `report` and `trees` leave out a copy that was deleted after the scan. Each - checks only the copy itself with `lstat`: the file, or the directory at the - top of a tree. A group left with one copy is not reported. A missing `scan` - operand is still a fatal error (2026-11-11, - https://git.eeqj.de/sneak/sfdupes/issues/173) -- The usage line of `scan` no longer ends in `[flags]`. It is the line - `README.md` shows under "Subcommands" (2026-11-11, - https://git.eeqj.de/sneak/sfdupes/issues/169) -- `report` and `trees` leave out empty files. `report` writes no group of size - 0, and `trees` writes no group whose trees hold 0 bytes. An empty file still - counts toward a tree's digest (2026-11-11, - https://git.eeqj.de/sneak/sfdupes/issues/178) -- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the - command, and it calls `Main` in the package `internal/sfdupes`. `main.go` - became `internal/sfdupes/cli.go`. Every other file kept its name. The install - path is now `sneak.berlin/go/sfdupes/cmd/sfdupes` (2026-11-10, - https://git.eeqj.de/sneak/sfdupes/issues/157) - -- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then - what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees` - prints the report again (2026-11-06, - https://git.eeqj.de/sneak/sfdupes/issues/154) - -- `scan` resolves each operand to its real path, symlinks followed, before the - walk. A symlink operand is scanned at its target instead of skipped. With - `/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub` - walks `sub` on its own instead of dropping it and deleting its records - (2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144) - -- `README.md` and a comment in `internal/sfdupes/scan.go` no longer say ZFS - counts a new file's blocks only seconds after the write. The count can land in - under a second. Both now say up to seconds after the write, as the comment on - `diskSizeOf` in `internal/sfdupes/scan_test.go` does (2026-11-03, - https://git.eeqj.de/sneak/sfdupes/issues/152) - -- The three disk size tests in `internal/sfdupes/scan_test.go` no longer fail at - random on ZFS. `unix.Sync` can return before ZFS has counted a file's blocks, - and one ZFS commit can take a file through several sizes. The test that needs - the size to change waits for it, for at most 30 seconds. Each test reads the - size before and after its scan. The rewrite test rescans until the two reads - match, then compares exactly. The other two fail only on a `disk_size` outside - the two reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150) - -- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by - sentence. Every completed step is kept, with its date and its issue or branch. - The policy-compliance checklist of 2026-07-22 is folded into its step - (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135) - -- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads - it. It leads with what the program is and how to use it. A reference section - covers every setting and every subcommand. Development and testing come last. - Every statement was checked against the code on `next`. Five comments in - `internal/sfdupes/cli_test.go` cite "Settings", where their rules now live. - One comment each in these files cites what the README now says: - - `internal/sfdupes/scan_test.go`; - - `internal/sfdupes/trees_test.go`; - - `internal/sfdupes/exclude_test.go`. - - (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135) - -- Each record holds `disk_size`, the bytes the file takes on disk. That is its - allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already - makes. `trees` prints a `disk_size` column after `size`. It is the sum over - the `dupe` copy's files. On a compressed or sparse file `disk_size` differs - from `size`. A rescan updates the `disk_size` of an unchanged file whose size - on disk changed, without reading the file (2026-10-27, - https://git.eeqj.de/sneak/sfdupes/issues/147) - -- The database keeps the number of records and the sum of their sizes in the one - row of a `totals` table. Three triggers on `files` keep that row current. - `info` reads that row. So do `scan`'s `load` phase and the `read` phases of - `report` and `trees`, for their totals. No count reads every record any more. - The triggers make each write to `files` slower. `scan` keeps SQLite's - temporary data in memory. That wins back part of that time (2026-10-27, - https://git.eeqj.de/sneak/sfdupes/issues/146) - -- an operand at or inside a directory named `.zfs` is never scanned, whether - given alone or under another operand. `scan` warns about it, counts it as - skipped and does not walk it (2026-10-26, - https://git.eeqj.de/sneak/sfdupes/issues/142) - -- A file or directory removed while the scan runs is not an error, in the walk - or in the hash phase. The scan prints no line for it and does not count it as - skipped. When the first of several hard links to a file is gone, the hash - phase reads the next one instead. On a terminal every progress display starts - with the spinner, the hash phase's included. A phase with an exact total ends - its line with its percent and a 10-column bar at the terminal's right edge. - The bar no longer moves as the line's length changes. A line too wide for the - terminal now loses its rate before its bar. The time remaining is labelled - `remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133) - -- The `head`, `tail` and `content` columns are BLOBs, each holding the raw 32 - bytes of a SHA-256. That is half the size of the hex text they held. The - hashes stay raw from the hash phase through the database to `report` and - `trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26, - https://git.eeqj.de/sneak/sfdupes/issues/134) - -- The hash phase gives three hashes to every file that the walk records. `head` - and `tail` are the SHA-256 of its first and last 64 KiB. `content` is the - SHA-256 of the whole file under 100 MiB, and of 1 MiB at each gigabyte from - 100 MiB up. Gone are: - - the 10 MiB and 50 MiB thresholds; - - the size census, which counted the sizes of every walked file and of every - record outside the operands. A file was hashed only when the census showed - another file of its size; - - the records without hashes; - - the content phase. - - A scan has four phases, and a rescan reads only new and changed files. - `info` no longer counts the records with `head` and `tail` hashes, nor those - with a content hash, since every record has all three (2026-10-25, - https://git.eeqj.de/sneak/sfdupes/issues/132) - -- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` were copied - again from `sneak/prompts` `next` at `0653579`. Two steps now come before - `ARG CACHEBUST`: - - the module download in the `lint`, `test` and `builder` stages of the - `Dockerfile`; - - the package install in `builder`. - - Docker keeps them until one of these changes: - - the base image; - - the install line; - - `go.mod` or `go.sum`. - - Every step that checks or compiles the code still comes after - `ARG CACHEBUST`. `golangci-lint run` also keeps the linter's own cache. - README says the same (2026-10-18, - https://git.eeqj.de/sneak/sfdupes/issues/109) - -- The release rule was set. Until 1.0, each tip of `main` that passes the build - gets the next `v0.0.N` tag. - `go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest` then installs it. - `go.mod` retracts `v0.0.1`, the early build without a database. When the - linker set no version, `sfdupes --version` prints the version Go recorded in - the binary. That is the tag, or the pseudo-version of a `go install` build. It - prints `dev` only when neither is known. README "Install" documented: - - the rule; - - what `@main` and `v0.0.1` give; - - how to check an installed binary. - - (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/116) - -- `scan` creates the database's directory where the database path leads. The - directory is the path with its last element cut off as text, so a `..` in it - stays. A `..` after a symlinked directory therefore goes through the symlink, - as it does when the file is opened. With - `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a symlink to - `/data/x`, it now creates `/data/new` and the database in it. It used to - create `/srv/new` and then fail (2026-10-17, - https://git.eeqj.de/sneak/sfdupes/issues/119) - -- `scan` never records its own database. Once the database and its `.lock` file - are open, it notes the device and inode of each of these: - - the database file; - - its `-wal` and `-shm` files; - - the `.lock` file. - - The walk passes over any file with the device and inode of one of them, - however it reaches the file. It prints no line for such a file and counts - nothing. A record an earlier scan left for one of them is deleted - (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115) - -- `scan` never records or reads anything on a filesystem that is not persistent, - such as `tmpfs` or `proc`. Such a filesystem is one the kernel makes up or - keeps in memory. `scan` matches them by device ID, however the walk or the - content phase reaches a path. As the scan starts, it lists the mounted - filesystems of those types, from `/proc/self/mountinfo` on Linux and from the - mount list on macOS. No flag turns that off. Below an operand, the walk does - not enter the default excludes either. So `sfdupes scan /` skips the kernel's - view of the running system and the temporary directories. On Linux those are: - - ``` - /proc, /sys, /dev, /run, /tmp, /var/tmp - ``` - - On macOS they are: - - ``` - /dev, /private/tmp, /private/var/tmp, /private/var/vm - /System/Volumes/VM, /System/Volumes/Data - ``` - - A Windows list exists in the code but is not used. An operand at or inside a - default exclude is still walked. The scan prints a line for each path it - leaves out, without counting it as skipped. The records there count as - outside the scanned operands, as for `-x`. `--no-default-excludes` turns off - the default excludes only, and `scan --help` lists them (2026-10-17, - https://git.eeqj.de/sneak/sfdupes/issues/124) - -- A progress display's rate covers the items completed over the last 5 seconds. - When stderr is not a terminal, it covers the items since the line before. It - falls to 0 while nothing completes. A burst after a wait no longer reads as a - steady rate of over 1,000 files a second. The time remaining still comes from - the phase's average. Once no item has completed for 10 seconds, the line shows - `stalled` and where the phase waits in place of the rate, as - `stalled disk/db`. The places are: - - `disk`; - - `db`; - - `out`, for stdout; - - `cpu`. - - A phase that may wait in two places names both. Each phase's word is set in - the code that defines the phase. A line too wide for the terminal loses its - bar first, then its rate or those words. Four parts stay whole: - - the phase; - - the progress; - - the elapsed time; - - the ETA, the time remaining. - - The content phase had one display for its checks and one for its reads. Both - lost those words on an 80-column terminal, at counts like those of README's - example. That happened for most elapsed times and ETAs in minutes. Every - other phase kept them whenever the elapsed time and the ETA were both in - minutes. The content phase writes its last partial batch, the records not - yet written, under a display of its own (2026-10-16, - https://git.eeqj.de/sneak/sfdupes/issues/125) - -- New subcommand `info` prints the database's details on stdout, one per line as - a name, a tab and the value: - - the path and where it came from; - - the sizes of the database file and of its `-wal` file, when there is one; - - when the database file was last changed; - - the schema version; - - the number of records; - - how many records have `head` and `tail` hashes; - - how many records have a content hash; - - the sizes of all records added up. - - It opens the database as `report` does and shows two phases on stderr, - `[1/2] open` and `[2/2] count`. The schema version and the counts come from - one read-only transaction, a read of the database at one moment. The counts - come from one query (2026-10-16, - https://git.eeqj.de/sneak/sfdupes/issues/117) - -- `report` shows its work on stderr in two phases, `[1/2] open` and - `[2/2] read`, with the progress display `scan` and `trees` use. The display - runs from the moment `report` starts until the summary line. `open` says it is - opening the database, also while it waits on a busy one. `read` counts the - records, then reads them against that count in signature order. SQLite takes - that order from the `files_signature` index without sorting, so the first - record comes at once. `report` forms the duplicate groups itself and holds - only those of the size it is reading. It writes them ordered by first path - before it reads the next size. stdout and the summary line are unchanged, and - SQLite no longer needs a temporary file to sort in (2026-10-16, - https://git.eeqj.de/sneak/sfdupes/issues/114) - -- `trees` shows its work on stderr in four phases, from `[1/4] open` to - `[4/4] write`, each with the progress display `scan` uses. The display runs - from the moment `trees` starts until the summary line. `open` says it is - opening the database, also while it waits on a busy one. `read` counts the - records, then reads them against that count. `group` counts the directories it - groups, and `write` the groups it writes. A display's rate names what its - phase counts: - - files; - - records; - - directories; - - groups. - - stdout and the summary line are unchanged. When stdout and stderr are the - same file, as one terminal is, no line of stdout shares a line with the - display. When they are not, the display keeps being drawn while stdout waits - on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113) - -- Each `scan` phase's progress display names the phase and its place among the - five, as `[2/5] walk`. It runs from the moment that the phase starts. It - shows: - - the progress; - - the elapsed time; - - the ETA, the time remaining. - - `load` counts the records first. The walk's total is estimated from the - records under the operands. The content phase takes its total from SQLite's - grouping alone. Until the first record is returned, it says it is searching - the database. On a terminal a phase with an exact total draws its line - followed by the bar of the progress-bar library. Any other phase draws a - spinner followed by the line. Every display is drawn again several times a - second. No line reaches the terminal's last column, however narrow the - terminal is, even after it is made narrower during a phase. A line too wide - leaves out its rate, then its bar. The rest stays whole. The line is cut - short only when the rest alone does not fit, as below about 70 columns late - in a large scan. When stderr is not a terminal, the line is printed every 5 - seconds whether or not an item completed (2026-10-16, - https://git.eeqj.de/sneak/sfdupes/issues/112) - -- `scan`, `report` and `trees` first print a line on stderr naming the database - they use, as an absolute path. The line says whether that path is the default - or comes from `SFDUPES_DATABASE`. `scan` says when the database is new. README - "Database" says when `scan` writes each file (2026-10-16, - https://git.eeqj.de/sneak/sfdupes/issues/111) - -- The canonical files and model scripts were copied again from `sneak/prompts` - `next` at `e3f79c4`: - - `golangci-lint` v2.14.0; - - lint and test are phases of the `Dockerfile` that write no image, and - `make test` runs the suite under the race detector. Gone are - `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race`; - - every `docker build` in `script/` passes a new `CACHEBUST` value. Every - stage of the `Dockerfile` declares it first, so every `RUN` step runs on - every build. The steps that compile keep Go's build cache, and the tests - run with `-count=1`; - - `prettier` runs on the host, from the `node` and `yarn` that - `script/bootstrap` installs. The `prettier` and `markdown` stages are - gone, and the build stage installs `git` and `make` itself; - - a new push cancels the workflow's older run on the same branch, and a run - stops after 20 minutes; - - a coding agent's settings file is deleted. - - Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before - `ARG CACHEBUST`: - - the module download in `lint`, `test` and the build stage; - - the install of `git` and `make` in the build stage. - - Docker keeps them between builds. Every step that checks or compiles the - code still runs on every build (2026-10-15, - https://git.eeqj.de/sneak/sfdupes/issues/95) - -- With `-x`, an operand under another operand is no longer dropped when a - directory between the two, the inner one included, is on a different - filesystem than the outer one. The inner operand is then walked on its own. - For a regular-file operand the check starts at its directory. The records of - files gone from below it are deleted (2026-10-14, - https://git.eeqj.de/sneak/sfdupes/issues/106) - -- With `-x`, `scan` keeps the records below a directory on another filesystem - under its operand, instead of deleting them. It counts their sizes in the size - census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99) - -- The hash and content phases open each file without blocking and check that it - is still a regular file. A file that is now a FIFO or device node is warned - about and skipped instead of hanging the scan (2026-10-14, - https://git.eeqj.de/sneak/sfdupes/issues/100) - -- The scan lock file is named from the database's path with symlinks resolved, - after `scan` creates the database file if it is missing. A scan through a - symlink to the database therefore fails while another scan holds the lock - (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/101) - -- On a terminal, a warning for the last file of a phase no longer erases the - phase's finished bar. The bar is drawn again below the warning (2026-10-14, - https://git.eeqj.de/sneak/sfdupes/issues/102) - -- `scan` records the mtime to the nanosecond, as whole seconds in `mtime` plus - `mtime_nsec`, and compares it at that resolution. A same-size rewrite within - the same second is hashed again (2026-10-07, - https://git.eeqj.de/sneak/sfdupes/issues/12) - -- The narration was cut from `TODO.md` Completed Steps and from the comments in - `script/`, `Dockerfile` and `Dockerfile.lint` (gone since - https://git.eeqj.de/sneak/sfdupes/issues/95). The Workflow section now - branches from and merges to `next` (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/49) - -- `make test-race` ran the test suite under the race detector in a cgo-enabled - container, outside `make check`. Since - https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite - under the race detector, in the `Dockerfile`'s Debian-based `test` phase. - `make test-race` is gone (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/18) - -- A bare `docker build .` failed, naming `script/cibuild` and `script/docker`, - rather than serve the gates, the lint and test steps, from cache. Since - https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as - `REPO_POLICIES.md` requires, and may serve the gates from cache. The builds in - `script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/39) - -- `make fmt` and `make fmt-check` run `prettier` over all Markdown, and the - Gitea workflow checks it. All Markdown was reformatted (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/19) - -- `script/lint` writes no image, so a run no longer leaves an untagged one - behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48) - -- Tests cover: - - a missing database; - - `scan` keeping stdout empty; - - its skip warning; - - the `report` and `trees` summary lines; - - every subcommand going through `runE`. - - (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16) - -- `.golangci.yml` was replaced with the current canonical copy, which uses - `gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/26) - -- A test fails when either `hashWorker` cancellation check in - `internal/sfdupes/scan.go` is removed (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/83) - -- A database path holding `?`, `#` or `%` opens exactly the file it names - (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55) - -- `scan` rejects `--workers` below 1 as a usage error instead of running - single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10) - -- A test fails when either walk cancellation check in `internal/sfdupes/scan.go` - is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81) - -- A test checks that `scan` refuses a database with another schema version - (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64) - -- Four inaccurate comments in `internal/sfdupes/cancel_test.go` were corrected, - and `walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles` - (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/33) - -- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/17) - -- `scan` creates the schema in one transaction. A version-0 database with a - `files` table is refused with a clear schema-version error (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/11) - -- README documents install, Docker, a daily `cron` scan and how to read and - check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54) - -- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home - and copied the sources with `--chown`, so no `chown -R` walked them. Since - https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and - nothing changes owner. The build stage only compiles, as root, and the tests - run as `nobody` in the `test` phase (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/43) - -- `--version` prints `sfdupes VERSION` to stdout. README documents it and - `--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15) - -- `scan` stops cleanly on `SIGINT` or `SIGTERM`: it commits what it has hashed, - deletes nothing more, and exits 1 (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/5) - -- `report` and `trees` stream the records instead of holding them all in memory. - The schema gains the `files_signature` index (2026-10-04, - https://git.eeqj.de/sneak/sfdupes/issues/14) - -- Progress prints at once on a non-terminal and uses a real terminal test. - Warnings print through a spinner instead of racing its redraw (2026-10-03, - https://git.eeqj.de/sneak/sfdupes/issues/13) - -- These operands are warned about and skipped, and the records beneath them are - kept: - - symlinks; - - sockets; - - FIFOs; - - device nodes; - - `.zfs` directories. - - (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/9) - -- `scan` holds a lock on a lock file beside the database for its whole run, so a - second `scan` fails at once with exit 1 (2026-10-03, - https://git.eeqj.de/sneak/sfdupes/issues/53) - -- Tests cover stdout write failures in `report` and `trees`. README states that - `| head` ends `sfdupes` by `SIGPIPE` and that `>&-` writes to `/dev/null` - (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30) - -- `report` and `trees` open the database read-only, and `scan` leaves it out of - WAL mode, so reading needs only read access (2026-10-03, - https://git.eeqj.de/sneak/sfdupes/issues/8) - -- Four characters are escaped in `report`, `trees` and warning paths: - - tab; - - newline; - - carriage return; - - backslash. - - The root directory's path is `/` (2026-10-03, - https://git.eeqj.de/sneak/sfdupes/issues/7) - -- A plain `docker build .` stamps the `git` tag or short commit instead of - `dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage - stamps the `VERSION` build argument, else `git describe --tags --always`. It - fails if the context carries `.git` and the version is still empty, `dev` or - `unknown`. The Gitea workflow checks out the full history (`fetch-depth: 0`), - so it stamps the same value as `make build` (2026-10-02, branch `next`, - https://git.eeqj.de/sneak/sfdupes/issues/67) - -- The 1 KiB end-window sampling was replaced with the hashes chosen by file - size, and README "Duplicate detection" documented every size threshold. A file - under 10 MiB was hashed in full, and its `head`, `tail` and `content` all held - that hash. A larger file got only its 64 KiB `head` and `tail` in the hash - phase. The content phase ran after the update phase. It read a file for - `content` only when its size, `head` and `tail` matched another record's, from - this scan or an earlier one. `content` was the whole file below 50 MiB, and - gigabyte-spaced 1 MiB samples at or above. It never read a file gone or - changed since its record was written. `report` and `trees` left out any record - without a `content` hash. The `content` column is part of the version 1 - schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132 - (2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61) - -- The dead `files.dat` references were removed from `Makefile`, `.gitignore` and - `.dockerignore` (2026-09-21, branch `next`, - https://git.eeqj.de/sneak/sfdupes/issues/22) - -- The lint-image pin comments and `FROM` form in `Dockerfile` and - `Dockerfile.lint` were fixed. Both pins became the policy - `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`, - without the false `(Debian-based)` note or the tag. Since - https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase - holds the only `golangci-lint` pin, in that form. `Dockerfile.lint` is gone, - and so is `script/verify-lint-image-pin`, which compared the two pins - (2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/25) - -- All linting ran in Docker, via `Dockerfile.lint` and `script/lint`. Per the - owner ruling the linter is never installed on a host, and - `golangci-lint config verify` runs before `golangci-lint run`. - `script/bootstrap` stopped installing or pinning the linter, and - `script/verify-linter-pin` was retired. Since - https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the - `Dockerfile`'s `lint` phase. `script/lint` builds that phase alone, and the - build stage depends on it through `COPY --from=lint /src/go.sum /dev/null`. - `Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside - an image build may run `docker`, so the phase calls `golangci-lint` directly - (2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/46) - -- The Docker build stage's prerequisites were installed by running - `script/bootstrap` instead of `apk add --no-cache make` inline. - `script/verify-linter-pin` failed the build unless the linter copied from the - lint stage was the version that `script/bootstrap` pinned. Builds then took up - to 5m14s from an empty cache, mostly in a `chown -R` of the module cache, - filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since - https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and - `make` with `apk add --no-cache` and only compiles. The `lint` and `test` - phases are the gates (2026-08-09, branch `dockerfile-bootstrap`, - https://git.eeqj.de/sneak/sfdupes/issues/42) - -- The Docker layer cache is defeated for the gate steps, so `script/cibuild` and - `script/docker` cannot report a pass they did not earn. The `Dockerfile` - copies the tree before its gates. On an unchanged tree Docker served them from - cache, and the build exited 0 without running them. The fix was a - `CHECK_EPOCH` build argument. Since - https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/` - passes a new `CACHEBUST` value instead. Every stage of the `Dockerfile` - declares it before every step that checks or compiles the code. Run as root, - the tests fail `TestScanHardlinkRunFailsTogether`, because root reads through - the `chmod(0)` the test relies on. So the `test` phase runs them as `nobody` - (2026-08-09, branch `cibuild-cache-bust`, - https://git.eeqj.de/sneak/sfdupes/issues/32) - -- `script/bootstrap` checked the installed `golangci-lint` version instead of - only its presence. The version lived only in `GOLANGCI_LINT_VERSION`, with the - `go install` module reference derived from it. Any installed version that was - not the pin was reinstalled, whether: - - older; - - newer; - - absent; - - unparseable. - - `go install` writes into `GOBIN`, or `GOPATH/bin`, while `make lint` ran the - first `golangci-lint` on `PATH`. So bootstrap re-read the effective version - after installing. On a mismatch it printed both paths and both versions and - exited non-zero. It did not reorder `PATH` or delete anyone's binary. The - `--version` call kept its stderr and was bounded by `timeout(1)` where that - exists. `git`, `make` and `go` kept presence-only checks. The version check - was verified by bootstrapping this host from v2.10.1 to v2.12.2, and again - to a no-op. Stub runs of the script under `dash` covered: - - a thirteen-input version-parse matrix; - - a shadowed install, which must exit non-zero; - - an install destination not on `PATH`; - - `GOBIN` set; - - a wedged binary, which must hit the timeout. - - `make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been - hiding findings on `main`. Superseded by - https://git.eeqj.de/sneak/sfdupes/issues/46, since the linter is no longer - installed on a host (2026-08-09, branch `bootstrap-version-check`, - https://git.eeqj.de/sneak/sfdupes/issues/24) - -- The hash worker pool unwinds on the error path. The pool is an owned, - context-aware `hashPool`. Every blocking send in the feeder and the workers - selects on `ctx.Done()`, and `jobs` is closed on every path out. `hashPhase` - defers `pool.stop()`, which cancels and then drains `results` until the last - goroutine has exited. Draining is what frees a worker parked on a send. `ctx` - is passed from `cmd.Context()` as the first parameter, through: - - `runScan`; - - `syncScan`; - - both worker pools; - - the whole database layer. - - The walk pool gets the same treatment, plus a `ctx.Err()` guard after the - walk. A cancelled walk yields a partial size census, and every file that it - never reached looks vanished to the update phase. That phase's own `BeginTx` - also fails on the cancelled context before deleting anything. The guard is - still the barrier once an interrupted scan may commit what it has. Tests - drive `run(scan)` against a database whose insert trigger aborts. They - assert that the scan fails instead of hanging, and that the goroutine count - returns to its pre-scan baseline. Other tests cancel a scan part-way through - the walk, deterministically, by counting the scan's own checks of - `ctx.Done()`. They assert that it stops at the guard with every record - intact. It then holds a partial size census and a still-populated record - index. Direct tests cover the remaining cancellation branches of both pools. - They test: - - `sendEvent`; - - the walk workers; - - `dispatchDirs`; - - `feedHashJobs`; - - `hashWorker`; - - `hashPhase`. - - (2026-08-09, branch `hash-pool-cleanup`, - https://git.eeqj.de/sneak/sfdupes/issues/6) - -- The database is closed on every fatal exit path. `fatalf` and its `os.Exit(1)` - are gone. So the deferred `db.Close()` runs when a subcommand fails, and with - it the SQLite WAL checkpoint. These return errors instead: - - `runScan`; - - `runReport`; - - `runTrees`; - - `loadRecords`; - - `resolveRoots`. - - The single exit point is `run` in `internal/sfdupes/cli.go`. It maps a - `fatalError`, anything a subcommand returned, to exit 1, and cobra's own - argument and flag errors to exit 2. So a runtime failure is never reported - as a usage error, nor printed with the usage text. - `internal/sfdupes/cli_test.go` drives the CLI in-process and asserts the - exit codes from README "Error handling" and the stdout/stderr split. It also - checks that a fatal error after the database is open leaves no `-wal` or - `-shm` file behind, for `scan`, `report` or `trees` (2026-08-09, branch - `db-close-on-fatal`, https://git.eeqj.de/sneak/sfdupes/issues/4) - -- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as - `38a01bd`. The pinned linter in the `Dockerfile` lint stage and - `script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was - replaced with the canonical file. Four thresholds now live under - `linters.settings`, as the v2 schema requires, so they are applied: - - `lll`; - - `funlen`; - - `cyclop`; - - `dupl`. - - No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`, - https://git.eeqj.de/sneak/sfdupes/issues/3) - -- `Makefile` targets were converted to scripts-to-rule-them-all `script/` - entrypoints, like the other managed repos. All 12 entrypoints exist: - - `bootstrap`, `setup` and `projectname`; - - `test`, `lint`, `fmt`, `fmt-check` and `check`; - - `docker`, `cibuild`, `precommit` and `install-precommit`. - - Every `Makefile` target does nothing but call its script (2026-07-26, commit - `3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1) - -- The binary is the default `make` target. Plain `make` builds `sfdupes`, where - it used to run `check` plus `build`. `make build` remains as an alias - (2026-07-24, branch `make-default-target`) - -- The phases became scan-wide, with concurrent operands and batched updates. All - operands seed the shared walk pool, and every pass runs once over the whole - scan. So the totals and ETAs cover the whole scan. The per-operand walk, hash - and update cycles and their stderr announcements are gone. The update pass - writes in batches, each in its own transaction. The filesystem is - authoritative and the database follows it, so one scan's changes need not land - all at once (2026-07-24, branch `scan-wide-phases`) - -- The stat pass was split back out of the walk. Phases are strictly sequential - again, per operand, with parallelism only inside each phase: - - walk; - - stat; - - hash; - - update. - - The walk enumerates paths with per-directory workers, and the stat pass - lstats them with per-file workers. That restores the exact total and ETA of - the stat bar (2026-07-24, branch `parallel-phases`) - -- Each operand is announced on stderr before its passes. With per-operand walk, - hash and update cycles, a multi-operand run such as `scan /srv/*` showed pass - totals that looked like the whole run's (2026-07-24, branch - `scan-operand-progress`) - -- The walk became parallel. The walk pass was a single goroutine and took hours - at about 20 million files on a busy pool. One ZFS server took 4 hours for 22 - million files. It is now a walk by a pool of per-directory workers that - records size and mtime as it goes. That folds away the separate stat pass and - halves the metadata reads. Each `PATH` operand commits in its own transaction, - so an interrupted scan keeps completed operands (2026-07-24, branch - `parallel-walk`) - -- `scan` gained a persistent database. It maintains an SQLite database that - survives between runs, with the absolute path as each record's key. The driver - is `modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only - new or changed files, by mtime and size. It deletes the records of files - vanished from under the scanned operands and leaves records outside them - untouched. So `scan` can run daily from `cron`. `report` and `trees` read the - database, with no positional arguments, instead of a scan stream. The database - is at `/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL - journaling plus a single-transaction update keep a report run during a scan - safe (2026-07-24, branch `persistent-database`) - -- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, was added. `v0.0.1` - was tagged, and `main` and the tags were pushed (2026-07-23) - -- The `scan` command line was reworked. Required `PATH...` operands, parsed by - cobra, replace the `/srv` `-root` default. The new `-x`, `--one-file-system` - flag follows the GNU convention and stops at filesystem boundaries, which the - walk crosses by default (2026-07-23, branch `scan-required-paths`) - -- The repo was brought into compliance with `REPO_POLICIES.md` (the version of - 2026-07-06), the existing repo checklist and the Go styleguide. The audit of - 2026-07-22 found these gaps, and the branch closed them: - - `.gitignore`, with the standard patterns for secrets and for OS, editor - and Go files. The compiled `sfdupes` binary and `files.dat` had been - sitting untracked in the tree; - - `.editorconfig`; - - `LICENSE` and the README License section. MIT was assumed from house - convention, for the owner to confirm; - - `REPO_POLICIES.md` in the repo root; - - the canonical `.golangci.yml`, with its 150 findings fixed, so that - `make lint` is clean; - - the `Makefile` targets the policy requires: - - `test` and `lint`; - - `fmt` and `fmt-check`; - - `docker` and `hooks`. - - `check` depended on `build`, which wrote the binary. `make check` must - not modify files, so `check` no longer depends on `build`; - - - a test suite, which the policy required with a 30-second timeout and the - conditional `-v` rerun. At 64% coverage, it covers: - - parsing and grouping; - - digests and suppression; - - hashing and the scan pipeline; - - a multistage `Dockerfile` and `.dockerignore`. The `Dockerfile` had: - - hash-pinned images; - - a lint stage that failed fast; - - a build stage that ran `make check`; - - `.gitea/workflows/check.yml`, running `docker build .` on push with the - checkout action pinned by commit SHA; - - the README sections the policy requires. The first is a Description whose - first line names the project, its purpose and category, its license and - its author. Then come Getting Started and Rationale, and at the end TODO, - License and Author; - - the stale README non-goal "no git repository setup and no CI" removed; - - the pre-commit hook, installed with `make hooks` once that target existed. - - The code was already `gofmt`-clean, so no standalone formatting commit was - needed (2026-07-23, branch `repo-policy-compliance`) - -- `git init` was run, with a README-only first commit. The code baseline was - committed on `main` (2026-07-22) - -- The `scan`, `report` and `trees` subcommands were implemented (pre-git - history) - # Future Steps - Possible later features, out of scope per README, are a byte-for-byte check of