Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c64a3c416d |
@@ -39,7 +39,8 @@ are known.
|
||||
it. It leads with what the program is and how to use it. A reference section
|
||||
covers the settings, the database, the output formats and the exit codes;
|
||||
development and testing come last. Every statement was checked against the
|
||||
code on `next` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
code on `next`. Three comments in `main_test.go` cite "Settings", where their
|
||||
rules now live (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
|
||||
- A file or directory removed while the scan runs is not an error, in the walk
|
||||
or in the hash phase. The scan prints no line for it and does not count it as
|
||||
@@ -62,7 +63,9 @@ are known.
|
||||
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
|
||||
up. Gone are:
|
||||
- the 10 MiB and 50 MiB thresholds;
|
||||
- the size census;
|
||||
- the size census: the sizes of every walked file and of every record
|
||||
outside the operands, by which a file was hashed only when another file
|
||||
shared its size;
|
||||
- the records without hashes;
|
||||
- the content phase.
|
||||
|
||||
@@ -107,20 +110,20 @@ are known.
|
||||
|
||||
- `scan` never records or reads anything on a filesystem that is not persistent:
|
||||
`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types. It
|
||||
matches them by device ID. As the scan starts, it lists the mounted
|
||||
filesystems of those types, from `/proc/self/mountinfo` on Linux and from the
|
||||
mount list on macOS. No flag turns that off. Below an operand, the walk does
|
||||
not enter the default excludes either, so `sfdupes scan /` skips the kernel's
|
||||
view of the running system and the temporary directories. On Linux those are
|
||||
`/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`. On macOS they are
|
||||
`/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
|
||||
`/System/Volumes/VM` and `/System/Volumes/Data`. A Windows list exists in the
|
||||
code but is not used. An operand at or inside a default exclude is still
|
||||
walked. The scan prints a line for each path it leaves out, without counting
|
||||
it as skipped, and the records there count as outside the scanned operands, as
|
||||
for `-x`. `--no-default-excludes` turns off the default excludes only, and
|
||||
`scan --help` lists them (2026-10-17,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
||||
matches them by device ID, however the walk or the content phase reaches a
|
||||
path. As the scan starts, it lists the mounted filesystems of those types,
|
||||
from `/proc/self/mountinfo` on Linux and from the mount list on macOS. No flag
|
||||
turns that off. Below an operand, the walk does not enter the default excludes
|
||||
either, so `sfdupes scan /` skips the kernel's view of the running system and
|
||||
the temporary directories. On Linux those are `/proc`, `/sys`, `/dev`, `/run`,
|
||||
`/tmp` and `/var/tmp`. On macOS they are `/dev`, `/private/tmp`,
|
||||
`/private/var/tmp`, `/private/var/vm`, `/System/Volumes/VM` and
|
||||
`/System/Volumes/Data`. A Windows list exists in the code but is not used. An
|
||||
operand at or inside a default exclude is still walked. The scan prints a line
|
||||
for each path it leaves out, without counting it as skipped, and the records
|
||||
there count as outside the scanned operands, as for `-x`.
|
||||
`--no-default-excludes` turns off the default excludes only, and `scan --help`
|
||||
lists them (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/124)
|
||||
|
||||
- The rate a progress display shows covers the items completed over the last 5
|
||||
seconds, or since the line before when stderr is not a terminal. It falls to 0
|
||||
@@ -129,14 +132,15 @@ are known.
|
||||
phase's average. Once no item has completed for 10 seconds, the word `stalled`
|
||||
and where the phase waits take the rate's place, as `stalled disk/db`. The
|
||||
places are `disk`, `db`, `out` for stdout and `cpu`, or two of them for a
|
||||
phase that may wait on either. Each phase gives its word where it is defined.
|
||||
A line too wide for the terminal loses its bar first, then its rate or those
|
||||
words, so the phase, progress, elapsed time and ETA stay whole. On an
|
||||
80-column terminal, at counts like those of README's example, the content
|
||||
phase's checks and reads lose those words at most times in minutes. Every
|
||||
other phase keeps them whenever the elapsed time and the ETA are both in
|
||||
minutes. The content phase's last partial batch is written under a display of
|
||||
its own (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||
phase that may wait on either. Each phase's word is set in the code that
|
||||
defines the phase. A line too wide for the terminal loses its bar first, then
|
||||
its rate or those words, so the phase, progress, elapsed time and ETA stay
|
||||
whole. On an 80-column terminal, at counts like those of README's example, the
|
||||
content phase's `checks` and `reads` lines were too wide for those words. That
|
||||
held at most elapsed times and ETAs in minutes. Every other phase kept them
|
||||
whenever the elapsed time and the ETA were both in minutes. The content
|
||||
phase's last partial batch is written under a display of its own (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||
|
||||
- New subcommand `info` prints the database's details on stdout, one per line as
|
||||
a name, a tab and the value:
|
||||
@@ -189,8 +193,8 @@ are known.
|
||||
narrow, also after the terminal is made narrower during a phase. A line too
|
||||
wide leaves out its rate, then its bar, keeping the phase, progress, elapsed
|
||||
time and ETA whole. It is cut short only when those alone do not fit, as below
|
||||
about 70 columns late in a large scan. In plain mode the line is printed every
|
||||
5 seconds whether or not an item completed (2026-10-16,
|
||||
about 70 columns late in a large scan. When stderr is not a terminal, the line
|
||||
is printed every 5 seconds whether or not an item completed (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||
|
||||
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
||||
@@ -314,8 +318,8 @@ are known.
|
||||
`files` table is refused with a clear schema-version error (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/11)
|
||||
|
||||
- README documents install, Docker, a daily cron scan and how to read and check
|
||||
the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
|
||||
- README documents install, Docker, a daily `cron` scan and how to read and
|
||||
check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
|
||||
|
||||
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
||||
and copied the sources with `--chown`, so no `chown -R` walked them. Since
|
||||
@@ -359,12 +363,12 @@ are known.
|
||||
and warning paths. The root directory's path is `/` (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
||||
|
||||
- A plain `docker build .` stamps the git tag or short commit instead of `dev`.
|
||||
`.dockerignore` sends `.git` without `.git/config`. The build stage stamps the
|
||||
`VERSION` build argument, else `git describe --tags --always`. It fails if the
|
||||
context carries `.git` and the version is still empty, `dev` or `unknown`. CI
|
||||
checks out the full history (`fetch-depth: 0`), so it stamps the same value as
|
||||
`make build` (2026-10-02, branch `next`,
|
||||
- A plain `docker build .` stamps the `git` tag or short commit instead of
|
||||
`dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage
|
||||
stamps the `VERSION` build argument, else `git describe --tags --always`. It
|
||||
fails if the context carries `.git` and the version is still empty, `dev` or
|
||||
`unknown`. CI checks out the full history (`fetch-depth: 0`), so it stamps the
|
||||
same value as `make build` (2026-10-02, branch `next`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/67)
|
||||
|
||||
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
|
||||
@@ -438,11 +442,20 @@ are known.
|
||||
installing. On a mismatch it printed both paths and both versions and exited
|
||||
non-zero; it did not reorder `PATH` or delete anyone's binary. The `--version`
|
||||
call kept its stderr and was bounded by `timeout(1)` where that exists. `git`,
|
||||
`make` and `go` kept presence-only checks. `make check` and `make lint` were
|
||||
clean at v2.12.2, so v2.10.1 had not been hiding findings on `main`.
|
||||
Superseded by https://git.eeqj.de/sneak/sfdupes/issues/46: the linter is no
|
||||
longer installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
||||
`make` and `go` kept presence-only checks. It was verified by bootstrapping
|
||||
this host from v2.10.1 to v2.12.2, and again to a no-op. Stub runs of the
|
||||
script under `dash` covered:
|
||||
- a thirteen-input version-parse matrix;
|
||||
- a shadowed install, which must exit non-zero;
|
||||
- an install destination not on `PATH`;
|
||||
- `GOBIN` set;
|
||||
- a wedged binary, which must hit the timeout.
|
||||
|
||||
`make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been
|
||||
hiding findings on `main`. Superseded by
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/46: the linter is no longer
|
||||
installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
||||
|
||||
- The hash worker pool unwinds on the error path. The pool is an owned,
|
||||
context-aware `hashPool`. Every blocking send in the feeder and the workers
|
||||
@@ -454,12 +467,14 @@ are known.
|
||||
walk pool gets the same treatment, plus a `ctx.Err()` guard after the walk. A
|
||||
cancelled walk yields a partial size census, and every file it never reached
|
||||
looks vanished to the update phase. That phase's own `BeginTx` also fails on
|
||||
the cancelled context before deleting anything, but the guard is the barrier
|
||||
that still holds once an interrupted scan may commit what it has. Tests drive
|
||||
`run(scan)` against a database whose insert trigger aborts, and assert that
|
||||
the scan fails instead of hanging and that the goroutine count returns to its
|
||||
pre-scan baseline. Other tests cancel a scan part-way through the walk and
|
||||
assert that it stops at the guard with every record intact. Direct tests of
|
||||
the cancelled context before deleting anything. The guard is still the barrier
|
||||
once an interrupted scan may commit what it has. Tests drive `run(scan)`
|
||||
against a database whose insert trigger aborts, and assert that the scan fails
|
||||
instead of hanging and that the goroutine count returns to its pre-scan
|
||||
baseline. Other tests cancel a scan part-way through the walk,
|
||||
deterministically, by counting the scan's own checks of `ctx.Done()`. They
|
||||
assert that it stops at the guard, holding a partial size census and a
|
||||
still-populated record index, with every record intact. Direct tests of
|
||||
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
|
||||
and `hashPhase` cover the remaining cancellation branches of both pools
|
||||
(2026-08-09, branch `hash-pool-cleanup`,
|
||||
@@ -469,13 +484,13 @@ are known.
|
||||
are gone, so the deferred `db.Close()`, and with it the SQLite WAL checkpoint,
|
||||
runs when a subcommand fails. `runScan`, `runReport`, `runTrees`,
|
||||
`loadRecords` and `resolveRoots` return errors instead. The single exit point
|
||||
is `run` in `main.go`. It maps a `fatalError` to exit 1 and cobra's own
|
||||
argument and flag errors to exit 2, so a runtime failure is never reported as
|
||||
a usage error or printed with the usage text. `main_test.go` drives the CLI
|
||||
in-process and asserts the exit codes from README "Error handling" and the
|
||||
stdout/stderr split. That includes a fatal error after the database is open
|
||||
leaving no `-wal` or `-shm` file behind, for `scan`, `report` or `trees`
|
||||
(2026-08-09, branch `db-close-on-fatal`,
|
||||
is `run` in `main.go`. It maps a `fatalError`, anything a subcommand returned,
|
||||
to exit 1, and cobra's own argument and flag errors to exit 2. So a runtime
|
||||
failure is never reported as a usage error, nor printed with the usage text.
|
||||
`main_test.go` drives the CLI in-process and asserts the exit codes from
|
||||
README "Error handling" and the stdout/stderr split. That includes a fatal
|
||||
error after the database is open leaving no `-wal` or `-shm` file behind, for
|
||||
`scan`, `report` or `trees` (2026-08-09, branch `db-close-on-fatal`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||
|
||||
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as
|
||||
@@ -486,17 +501,17 @@ are known.
|
||||
schema, so they are applied. No new lint findings surfaced (2026-08-09, branch
|
||||
`golangci-v2.12.2`, https://git.eeqj.de/sneak/sfdupes/issues/3)
|
||||
|
||||
- Makefile targets converted to scripts-to-rule-them-all `script/` entrypoints,
|
||||
like the other managed repos. All 12 entrypoints exist:
|
||||
- `Makefile` targets converted to scripts-to-rule-them-all `script/`
|
||||
entrypoints, like the other managed repos. All 12 entrypoints exist:
|
||||
- `bootstrap`, `setup` and `projectname`;
|
||||
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
|
||||
- `docker`, `cibuild`, `precommit` and `install-precommit`.
|
||||
|
||||
Every Makefile target is a thin shim over them (2026-07-26, commit
|
||||
Every `Makefile` target does nothing but call its script (2026-07-26, commit
|
||||
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
|
||||
|
||||
- The binary is the default Make target: plain `make` builds `sfdupes`, where it
|
||||
used to run `check` plus `build`. `make build` remains as an alias
|
||||
- The binary is the default `make` target: plain `make` builds `sfdupes`, where
|
||||
it used to run `check` plus `build`. `make build` remains as an alias
|
||||
(2026-07-24, branch `make-default-target`)
|
||||
|
||||
- Scan-wide phases, concurrent operands and batched updates. All operands seed
|
||||
@@ -558,9 +573,11 @@ are known.
|
||||
- `LICENSE` and the README License section. MIT was assumed from house
|
||||
convention, for the owner to confirm;
|
||||
- `REPO_POLICIES.md` in the repo root;
|
||||
- the canonical `.golangci.yml`, with the 150 findings it raised fixed;
|
||||
- the canonical `.golangci.yml`, with the 150 findings it raised fixed, so
|
||||
that `make lint` is clean;
|
||||
- the `Makefile` targets `test`, `lint`, `fmt`, `fmt-check`, `docker` and
|
||||
`hooks`, and a `check` target that no longer depends on `build`;
|
||||
`hooks`. `check` depended on `build`, which wrote the binary, and
|
||||
`make check` must not modify files, so `check` no longer depends on it;
|
||||
- a test suite, which the policy required with a 30-second timeout and the
|
||||
conditional `-v` rerun. It covers parsing, grouping, digests, suppression,
|
||||
hashing and the scan pipeline, at 64% coverage;
|
||||
@@ -572,7 +589,7 @@ are known.
|
||||
name, purpose, category, license and author, then Getting Started,
|
||||
Rationale, TODO, License and Author;
|
||||
- the stale README non-goal "no git repository setup and no CI" removed;
|
||||
- the pre-commit hook installed.
|
||||
- the pre-commit hook, installed with `make hooks` once that target existed.
|
||||
|
||||
The code was already `gofmt`-clean, so no standalone formatting commit was
|
||||
needed (2026-07-23, branch `repo-policy-compliance`)
|
||||
|
||||
Reference in New Issue
Block a user