# Workflow - take an issue from the `1.0.0` milestone on the tracker; work not yet on the tracker gets filed as an issue first - branch from `next` - do the work, with tests, in small focused commits - record it at the top of Completed Steps (`TODO.md` changes in the same commit as the work) - push the branch and open a PR against `next` whose title ends with ` (closes #N)` - an independent review gates each merge to `next`; every finding is addressed or explicitly rebutted on the PR - only the owner merges `next` to `main` # Status - pre-1.0 - the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues under the `1.0.0` milestone are what remains before the tag, and this file records history and process, not the queue # Next Step - take the next issue from the `1.0.0` milestone on the tracker: https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source of truth for what is left before 1.0.0. Individual issues are deliberately not restated here; a copy in this file drifts out of date the moment the tracker moves # Completed Steps - new subcommand `info` prints the database's details on stdout, one per line as a name, a tab and the value: its path and where it came from, the sizes of the database file and of its `-wal` file when there is one, when the database file was last changed, the schema version, the number of records, how many have head and tail hashes and how many a content hash, and the sizes of all records added up. It opens the database as `report` does, and shows two phases on stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come from one read-only transaction, the counts from one query (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/117) - `report` shows its work on stderr in two phases, `[1/2] open` and `[2/2] read`, with the progress display `scan` and `trees` use, from the moment it starts until the summary line: `open` says it is opening the database, also while it waits on a busy one; `read` counts the records, then reads them against that count in signature order, which SQLite takes from the `files_signature` index without sorting, so the first record comes at once. `report` forms the duplicate groups itself, holding only those of the size it is reading, and writes them ordered by first path before it reads the next size. stdout and the summary line are unchanged, and SQLite no longer needs a temporary file to sort in (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/114) - `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`, `[3/4] group` and `[4/4] write`, each with the progress display `scan` uses, from the moment it starts until the summary line: `open` says it is opening the database, also while it waits on a busy one; `read` counts the records, then reads them against that count; `group` counts the directories it groups, and `write` the groups it writes. A display's rate names what its phase counts: files, records, directories or groups. stdout and the summary line are unchanged. When stdout and stderr are the same file, as one terminal is, none of stdout's lines shares a line with the display; when they are not, the display keeps being drawn while stdout waits on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113) - each `scan` phase's progress display names the phase and its place among the five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed time and ETA: `load` counts the records first, the walk's total is estimated from the records under the operands, and the content phase counts the records it checks from SQLite's grouping alone, then says it is searching the database until the first one is returned. On a terminal a phase with an exact total draws its line followed by the library's bar, and any other a spinner followed by the line; every display is drawn again several times a second. No line reaches the terminal's last column, however narrow, also after the terminal is made narrower during a phase: a line too wide leaves out its rate, then its bar, keeping the phase, progress, elapsed time and ETA whole, and is cut short only when those alone do not fit, as below about 70 columns late in a large scan. In plain mode the line is printed every 5 seconds whether or not an item completed (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/112) - `scan`, `report` and `trees` first print a line on stderr naming the database they use, as an absolute path, and whether that path is the default or comes from `SFDUPES_DATABASE`; `scan` says when the database is new. README "Database" says when `scan` writes each file (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/111) - re-vendor the canonical files and model scripts from `sneak/prompts` `next` at `e3f79c4`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile` that write no image, and `make test` runs the suite under the race detector, so `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race` are gone; every `docker build` in `script/` passes a new `CACHEBUST` value, which every stage of the `Dockerfile` declares first, so every `RUN` step runs on every build, while the steps that compile keep Go's build cache and the tests run with `-count=1`; prettier runs on the host, from the node and yarn `script/bootstrap` installs, so the `prettier` and `markdown` stages are gone and the build stage installs `git` and `make` itself; a new push cancels the workflow's older run on the same branch, and a run stops after 20 minutes; `.claude/settings.json` is deleted (2026-10-15, https://git.eeqj.de/sneak/sfdupes/issues/95) - with `-x`, an operand under another operand is walked on its own instead of being dropped when a directory from it (from its directory, for a regular file) up to but not including that operand is on a different filesystem than that operand, whose walk then cannot reach it, and the records of files gone from below it are deleted (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/106) - with `-x`, `scan` keeps the records below a directory on another filesystem under its operand, and counts their sizes in the size census, instead of deleting them (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99) - the hash and content phases open each file without blocking and check that it is still a regular file, so one replaced by a FIFO or device node after it was statted is warned about and skipped instead of hanging the scan (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/100) - the scan lock file is named from the database's path with symlinks resolved, after `scan` creates the database file if it is missing, so a scan through a symlink to the database fails while another scan holds the lock (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/101) - on a terminal, a warning for the last file of a phase no longer erases the phase's finished bar: the bar is drawn again below the warning (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/102) - `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus `mtime_nsec`, and compares it at that resolution, so a same-size rewrite within the same second is re-hashed (2026-10-07, https://git.eeqj.de/sneak/sfdupes/issues/12) - cut the narration from `TODO.md` Completed Steps and from the comments in `script/`, `Dockerfile` and `Dockerfile.lint` (gone since https://git.eeqj.de/sneak/sfdupes/issues/95); §Workflow now branches from and merges to `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49) - `make test-race` ran the test suite under the race detector in a cgo-enabled container, outside `make check` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/18). Since https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite under the race detector, in the `Dockerfile`'s Debian-based `test` phase, and `make test-race` is gone - a bare `docker build .` failed, naming `script/cibuild` and `script/docker`, rather than serve the gates from cache (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/39). Since https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as `REPO_POLICIES.md` requires, and may serve the gates from cache; the builds in `script/` pass a new `CACHEBUST` value, so theirs always run - `make fmt` and `make fmt-check` run prettier over all Markdown, and CI checks it; all Markdown reformatted (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/19) - `script/lint` writes no image, so a run no longer leaves an untagged one behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48) - tests cover a missing database, `scan` keeping stdout empty, its skip warning, the `report` and `trees` summary lines, and every subcommand going through `runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16) - `.golangci.yml` replaced with the current canonical copy, which uses `gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/26) - a test fails when either `hashWorker` cancellation check in `scan.go` is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83) - a database path holding `?`, `#` or `%` opens exactly the file it names (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55) - `scan` rejects `--workers` below 1 as a usage error instead of running single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10) - a test fails when either walk cancellation check in `scan.go` is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81) - test that `scan` refuses a database with another schema version (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64) - correct four inaccurate comments in `cancel_test.go` and rename `walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/33) - test the `-x` filesystem-boundary rules in `subdirJob` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/17) - `scan` creates the schema in one transaction; a version-0 database with a `files` table is refused with a clear schema-version error (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/11) - README documents install, Docker, a daily cron scan and how to read and check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54) - the `Dockerfile` build stage kept the Go module cache out of `builder`'s home and copied the sources with `--chown`, so no `chown -R` walked them (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43). Since https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and nothing changes owner: the build stage only compiles, as root, and the tests run as `nobody` in the `test` phase - `--version` prints `sfdupes VERSION` to stdout; README documents it and `--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15) - `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has hashed, deletes nothing more, exits 1 (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/5) - `report` and `trees` stream the records instead of holding them all in memory; the schema gains the `files_signature` index (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/14) - progress prints at once on a non-terminal, uses a real terminal test, and prints warnings through a spinner instead of racing its redraw (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/13) - warn about and skip symlink, socket, FIFO, device and `.zfs` operands, keeping the records beneath them (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/9) - `scan` holds a lock on a lock file beside the database for its whole run, so a second `scan` fails at once with exit 1 (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/53) - test stdout write failures in `report` and `trees`; README states that `| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null` (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30) - `report` and `trees` open the database read-only, and `scan` leaves it out of WAL mode, so reading needs only read access (2026-10-03, closes https://git.eeqj.de/sneak/sfdupes/issues/8) - escape tabs, newlines, carriage returns and backslashes in report, trees and warning paths; the root directory's path is `/` (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/7) - stamp the git tag or short commit in a plain `docker build .` instead of `dev` (2026-10-02, branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git` without `.git/config`; the build stage stamps the `VERSION` build argument, else `git describe --tags --always`, and fails if the context carries `.git` and the version is still empty, `dev` or `unknown`. CI checks out the full history (`fetch-depth: 0`) so it stamps the same value as `make build`. - replace the 1 KiB end-window sampling with the head/tail plus content-hash ladder (2026-09-22, branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection" documents every rung. A file under 10 MiB is hashed in full, and its `head`, `tail` and `content` all hold that hash. A larger file gets only its 64 KiB `head` and `tail` in the hash phase; the content phase, after the update phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above) only when its size, `head` and `tail` match another record's from this scan or an earlier one, and never reads a file gone or changed since its record was written. `report` and `trees` leave out any record without a `content` hash. The `content` column is part of the version 1 schema. - remove the dead `files.dat` references from `Makefile`, `.gitignore` and `.dockerignore` (2026-09-21, branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/22) - fix the lint-image pin comments and `FROM` form in `Dockerfile` and `Dockerfile.lint` (2026-08-10, branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/25): both pins became the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`, without the false `(Debian-based)` note or the tag. Since https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase holds the only golangci-lint pin, in that form, so `Dockerfile.lint` and `script/verify-lint-image-pin`, which compared the two pins, are gone. - run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10, branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling the linter is never installed on a host, and `golangci-lint config verify` runs before `golangci-lint run`. `script/bootstrap` stopped installing or pinning the linter, and `script/verify-linter-pin` was retired. Since https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the `Dockerfile`'s `lint` phase, which `script/lint` builds alone and the build stage depends on through `COPY --from=lint /src/go.sum /dev/null`; `Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside an image build may run docker, so the phase calls `golangci-lint` directly. - install the Docker build stage's prerequisites by running `script/bootstrap` instead of `apk add --no-cache make` inline (2026-08-09, branch `dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42), with `script/verify-linter-pin` failing the build unless the linter copied from the lint stage was the version `script/bootstrap` pinned. Builds then took up to 5m14s cold, mostly in a `chown -R` of the module cache, filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and `make` with `apk add --no-cache` and only compiles; the `lint` and `test` phases are the gates - bust the Docker layer cache for the gate steps, so `script/cibuild` and `script/docker` cannot report a green they did not earn (2026-08-09, branch `cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the `Dockerfile` copies the tree before its gates, so on an unchanged tree Docker served them from cache and the build exited 0 having run nothing. The fix was a `CHECK_EPOCH` build argument; since https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/` passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile` declares first. Run as root, the tests fail `TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)` the test relies on, so the `test` phase runs them as `nobody` - check the installed golangci-lint version in `script/bootstrap` instead of only its presence (2026-08-09, branch `bootstrap-version-check`, closes https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in `GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and any installed version that is not the pin — older, newer, absent or unparseable — is reinstalled. `go install` writes into `GOBIN` (or `GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so bootstrap re-reads the effective version after installing and, on a mismatch, prints both paths and both versions and exits non-zero; it does not reorder `PATH` or delete anyone's binary. The `--version` call keeps its stderr and is bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep presence-only checks. Verified by bootstrapping this host from v2.10.1 to v2.12.2 and again to a no-op, and by stub runs of the script under `dash` covering a thirteen-input version-parse matrix, a shadowed install that must exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged binary that must hit the timeout; `make check` and `make lint` are clean at v2.12.2, so v2.10.1 was not hiding any findings on `main` - unwind the hash worker pool on the error path (2026-08-09, branch `hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the pool is now an owned, context-aware `hashPool`: every blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels and then drains `results` until the last goroutine has exited — draining is what frees a worker already parked on a send. `ctx` is threaded from `cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole database layer, as the first parameter everywhere. The walk pool gets the same treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size census, and every file it never reached looks vanished to the update phase. That phase's own `BeginTx` also fails on the cancelled context before deleting anything, but the guard is the barrier that still holds once an interrupted scan may commit what it has. Tests drive `run(scan)` against a database whose insert trigger aborts and assert that the scan fails instead of hanging and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others cancel a scan part-way through the walk, deterministically, by counting its own consultations of `ctx.Done()`, and assert that it stops at the guard holding a partial census and a still-populated record index, with every record intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation branches of both pools - guarantee the database is closed on every fatal exit path (2026-08-09, branch `db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4): `fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return errors instead. The single exit point is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned) to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a runtime failure from being reported as a usage error or printing the usage text. New `main_test.go` drives the CLI in-process and asserts the exit codes from README §Error handling plus the stdout/stderr split, including that a fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`, `report` or `trees` - update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`, closes https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced `.golangci.yml` with the canonical file — the linter settings (`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per the v2 schema, so they are actually applied; no new lint findings surfaced - convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints like the other managed repos (2026-07-26, commit `3abeacf`, closes https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`) and every Makefile target is now a thin shim over them - make the binary the default Make target (2026-07-24, branch `make-default-target`): plain `make` now builds `sfdupes` (previously it ran `check` plus `build`); `make build` remains as an alias - scan-wide phases, concurrent operands, batched updates (2026-07-24, branch `scan-wide-phases`): all operands seed the shared walk pool and every pass runs once over the whole scan, so totals and ETAs are scan-global; the per-operand walk/hash/update cycles and their stderr announcements are gone; the update pass commits in batched transactions — the filesystem is authoritative and the database an eventually-consistent reflection, so scan-level atomicity is not required - split the stat pass back out of the walk (2026-07-24, branch `parallel-phases`): phases are strictly sequential again — walk, stat, hash, update per operand — with parallelism only inside each phase; the walk enumerates paths with per-directory workers and the stat pass lstats them with per-file workers, restoring the exact total/ETA stat bar - announce each operand on stderr before its passes (2026-07-24, branch `scan-operand-progress`): with per-operand walk/hash/update cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the whole run's - parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single goroutine and took hours at ~20M files on a busy pool (observed: 22M files in 4h on a ZFS server); it is now a per-directory worker-pool traversal that records size/mtime during the walk (folding away the separate stat pass, halving metadata I/O), and each `PATH` operand commits in its own transaction so an interrupted scan keeps completed operands - persistent scan database (2026-07-24, branch `persistent-database`): `scan` now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays disabled) keyed by absolute path that survives between runs — a rescan hashes only new or changed files (by mtime/size), deletes records for files vanished from under the scanned operands, and leaves records outside them untouched, so `scan` can be cronned daily; `report` and `trees` read the database (no positional arguments) instead of a scan stream. Database at `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL journaling plus a single-transaction update keep a report run during a scan safe - add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`, and push `main` plus tags (2026-07-23) - `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required `PATH...` operands via cobra flags replacing the `/srv` `-root` default; new `-x`/`--one-file-system` flag (GNU convention) to stop at filesystem boundaries, which are crossed by default - bring the repo into full policy compliance (2026-07-23, branch `repo-policy-compliance`; checklist below) - `git init` with README-only first commit; code baseline committed on `main` (2026-07-22) - implement `scan`, `report`, and `trees` subcommands (pre-git history) # Future Steps - possible later features (explicitly out of scope per README): full-content verification of candidates, removal-script helpers # Repo Policy Compliance Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone formatting commit is needed. - [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns - [x] `.editorconfig` missing - [x] `LICENSE` missing and README has no License section (MIT assumed from house convention — user to confirm) - [x] `REPO_POLICIES.md` missing from repo root - [x] `.golangci.yml` missing (install canonical copy); code must then pass `make lint` (150 findings fixed; `make lint` is clean) - [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`, `docker`, `hooks`; `check` currently depends on `build`, which writes the binary (`make check` must not modify files) - [x] no tests — `go test ./...` has nothing to run; policy requires real tests with a 30-second timeout and the conditional `-v` rerun pattern (suite covers parsing, grouping, digests, suppression, hashing, and the scan pipeline; 64% coverage) - [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast lint stage, build stage running `make check` - [x] `.dockerignore` missing - [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout action pinned by commit SHA) - [x] README lacks required sections: Description first line (name/purpose/category/license/author), Getting Started, Rationale, TODO, License, Author - [x] README non-goal "no git repository setup and no CI" is stale now that the repo is under git with CI - [x] pre-commit hook not installed (`make hooks` once the target exists) Accepted divergences (no action): - flat single-package layout with `.go` files in the repo root — fine for a small single-binary tool per the Go styleguide; the tracker audit agrees