Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
16a57547b8 |
@@ -39,8 +39,7 @@ are known.
|
|||||||
it. It leads with what the program is and how to use it. A reference section
|
it. It leads with what the program is and how to use it. A reference section
|
||||||
covers the settings, the database, the output formats and the exit codes;
|
covers the settings, the database, the output formats and the exit codes;
|
||||||
development and testing come last. Every statement was checked against the
|
development and testing come last. Every statement was checked against the
|
||||||
code on `next`. Three comments in `main_test.go` cite "Settings", where their
|
code on `next` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||||
rules now live (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
|
||||||
|
|
||||||
- A file or directory removed while the scan runs is not an error, in the walk
|
- A file or directory removed while the scan runs is not an error, in the walk
|
||||||
or in the hash phase. The scan prints no line for it and does not count it as
|
or in the hash phase. The scan prints no line for it and does not count it as
|
||||||
@@ -63,9 +62,7 @@ are known.
|
|||||||
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
|
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
|
||||||
up. Gone are:
|
up. Gone are:
|
||||||
- the 10 MiB and 50 MiB thresholds;
|
- the 10 MiB and 50 MiB thresholds;
|
||||||
- the size census: the sizes of every walked file and of every record
|
- the size census;
|
||||||
outside the operands, by which a file was hashed only when another file
|
|
||||||
shared its size;
|
|
||||||
- the records without hashes;
|
- the records without hashes;
|
||||||
- the content phase.
|
- the content phase.
|
||||||
|
|
||||||
@@ -110,20 +107,20 @@ are known.
|
|||||||
|
|
||||||
- `scan` never records or reads anything on a filesystem that is not persistent:
|
- `scan` never records or reads anything on a filesystem that is not persistent:
|
||||||
`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types. It
|
`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types. It
|
||||||
matches them by device ID, however the walk or the content phase reaches a
|
matches them by device ID. As the scan starts, it lists the mounted
|
||||||
path. As the scan starts, it lists the mounted filesystems of those types,
|
filesystems of those types, from `/proc/self/mountinfo` on Linux and from the
|
||||||
from `/proc/self/mountinfo` on Linux and from the mount list on macOS. No flag
|
mount list on macOS. No flag turns that off. Below an operand, the walk does
|
||||||
turns that off. Below an operand, the walk does not enter the default excludes
|
not enter the default excludes either, so `sfdupes scan /` skips the kernel's
|
||||||
either, so `sfdupes scan /` skips the kernel's view of the running system and
|
view of the running system and the temporary directories. On Linux those are
|
||||||
the temporary directories. On Linux those are `/proc`, `/sys`, `/dev`, `/run`,
|
`/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`. On macOS they are
|
||||||
`/tmp` and `/var/tmp`. On macOS they are `/dev`, `/private/tmp`,
|
`/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
|
||||||
`/private/var/tmp`, `/private/var/vm`, `/System/Volumes/VM` and
|
`/System/Volumes/VM` and `/System/Volumes/Data`. A Windows list exists in the
|
||||||
`/System/Volumes/Data`. A Windows list exists in the code but is not used. An
|
code but is not used. An operand at or inside a default exclude is still
|
||||||
operand at or inside a default exclude is still walked. The scan prints a line
|
walked. The scan prints a line for each path it leaves out, without counting
|
||||||
for each path it leaves out, without counting it as skipped, and the records
|
it as skipped, and the records there count as outside the scanned operands, as
|
||||||
there count as outside the scanned operands, as for `-x`.
|
for `-x`. `--no-default-excludes` turns off the default excludes only, and
|
||||||
`--no-default-excludes` turns off the default excludes only, and `scan --help`
|
`scan --help` lists them (2026-10-17,
|
||||||
lists them (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/124)
|
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
||||||
|
|
||||||
- The rate a progress display shows covers the items completed over the last 5
|
- The rate a progress display shows covers the items completed over the last 5
|
||||||
seconds, or since the line before when stderr is not a terminal. It falls to 0
|
seconds, or since the line before when stderr is not a terminal. It falls to 0
|
||||||
@@ -132,15 +129,14 @@ are known.
|
|||||||
phase's average. Once no item has completed for 10 seconds, the word `stalled`
|
phase's average. Once no item has completed for 10 seconds, the word `stalled`
|
||||||
and where the phase waits take the rate's place, as `stalled disk/db`. The
|
and where the phase waits take the rate's place, as `stalled disk/db`. The
|
||||||
places are `disk`, `db`, `out` for stdout and `cpu`, or two of them for a
|
places are `disk`, `db`, `out` for stdout and `cpu`, or two of them for a
|
||||||
phase that may wait on either. Each phase's word is set in the code that
|
phase that may wait on either. Each phase gives its word where it is defined.
|
||||||
defines the phase. A line too wide for the terminal loses its bar first, then
|
A line too wide for the terminal loses its bar first, then its rate or those
|
||||||
its rate or those words, so the phase, progress, elapsed time and ETA stay
|
words, so the phase, progress, elapsed time and ETA stay whole. On an
|
||||||
whole. On an 80-column terminal, at counts like those of README's example, the
|
80-column terminal, at counts like those of README's example, the content
|
||||||
content phase's `checks` and `reads` lines were too wide for those words. That
|
phase's checks and reads lose those words at most times in minutes. Every
|
||||||
held at most elapsed times and ETAs in minutes. Every other phase kept them
|
other phase keeps them whenever the elapsed time and the ETA are both in
|
||||||
whenever the elapsed time and the ETA were both in minutes. The content
|
minutes. The content phase's last partial batch is written under a display of
|
||||||
phase's last partial batch is written under a display of its own (2026-10-16,
|
its own (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
|
||||||
|
|
||||||
- New subcommand `info` prints the database's details on stdout, one per line as
|
- New subcommand `info` prints the database's details on stdout, one per line as
|
||||||
a name, a tab and the value:
|
a name, a tab and the value:
|
||||||
@@ -193,8 +189,8 @@ are known.
|
|||||||
narrow, also after the terminal is made narrower during a phase. A line too
|
narrow, also after the terminal is made narrower during a phase. A line too
|
||||||
wide leaves out its rate, then its bar, keeping the phase, progress, elapsed
|
wide leaves out its rate, then its bar, keeping the phase, progress, elapsed
|
||||||
time and ETA whole. It is cut short only when those alone do not fit, as below
|
time and ETA whole. It is cut short only when those alone do not fit, as below
|
||||||
about 70 columns late in a large scan. When stderr is not a terminal, the line
|
about 70 columns late in a large scan. In plain mode the line is printed every
|
||||||
is printed every 5 seconds whether or not an item completed (2026-10-16,
|
5 seconds whether or not an item completed (2026-10-16,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||||
|
|
||||||
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
||||||
@@ -318,8 +314,8 @@ are known.
|
|||||||
`files` table is refused with a clear schema-version error (2026-10-04,
|
`files` table is refused with a clear schema-version error (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/11)
|
https://git.eeqj.de/sneak/sfdupes/issues/11)
|
||||||
|
|
||||||
- README documents install, Docker, a daily `cron` scan and how to read and
|
- README documents install, Docker, a daily cron scan and how to read and check
|
||||||
check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
|
the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
|
||||||
|
|
||||||
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
||||||
and copied the sources with `--chown`, so no `chown -R` walked them. Since
|
and copied the sources with `--chown`, so no `chown -R` walked them. Since
|
||||||
@@ -363,12 +359,12 @@ are known.
|
|||||||
and warning paths. The root directory's path is `/` (2026-10-03,
|
and warning paths. The root directory's path is `/` (2026-10-03,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
||||||
|
|
||||||
- A plain `docker build .` stamps the `git` tag or short commit instead of
|
- A plain `docker build .` stamps the git tag or short commit instead of `dev`.
|
||||||
`dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage
|
`.dockerignore` sends `.git` without `.git/config`. The build stage stamps the
|
||||||
stamps the `VERSION` build argument, else `git describe --tags --always`. It
|
`VERSION` build argument, else `git describe --tags --always`. It fails if the
|
||||||
fails if the context carries `.git` and the version is still empty, `dev` or
|
context carries `.git` and the version is still empty, `dev` or `unknown`. CI
|
||||||
`unknown`. CI checks out the full history (`fetch-depth: 0`), so it stamps the
|
checks out the full history (`fetch-depth: 0`), so it stamps the same value as
|
||||||
same value as `make build` (2026-10-02, branch `next`,
|
`make build` (2026-10-02, branch `next`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/67)
|
https://git.eeqj.de/sneak/sfdupes/issues/67)
|
||||||
|
|
||||||
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
|
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
|
||||||
@@ -442,19 +438,10 @@ are known.
|
|||||||
installing. On a mismatch it printed both paths and both versions and exited
|
installing. On a mismatch it printed both paths and both versions and exited
|
||||||
non-zero; it did not reorder `PATH` or delete anyone's binary. The `--version`
|
non-zero; it did not reorder `PATH` or delete anyone's binary. The `--version`
|
||||||
call kept its stderr and was bounded by `timeout(1)` where that exists. `git`,
|
call kept its stderr and was bounded by `timeout(1)` where that exists. `git`,
|
||||||
`make` and `go` kept presence-only checks. It was verified by bootstrapping
|
`make` and `go` kept presence-only checks. `make check` and `make lint` were
|
||||||
this host from v2.10.1 to v2.12.2, and again to a no-op. Stub runs of the
|
clean at v2.12.2, so v2.10.1 had not been hiding findings on `main`.
|
||||||
script under `dash` covered:
|
Superseded by https://git.eeqj.de/sneak/sfdupes/issues/46: the linter is no
|
||||||
- a thirteen-input version-parse matrix;
|
longer installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
||||||
- a shadowed install, which must exit non-zero;
|
|
||||||
- an install destination not on `PATH`;
|
|
||||||
- `GOBIN` set;
|
|
||||||
- a wedged binary, which must hit the timeout.
|
|
||||||
|
|
||||||
`make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been
|
|
||||||
hiding findings on `main`. Superseded by
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/46: the linter is no longer
|
|
||||||
installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
||||||
|
|
||||||
- The hash worker pool unwinds on the error path. The pool is an owned,
|
- The hash worker pool unwinds on the error path. The pool is an owned,
|
||||||
@@ -467,14 +454,12 @@ are known.
|
|||||||
walk pool gets the same treatment, plus a `ctx.Err()` guard after the walk. A
|
walk pool gets the same treatment, plus a `ctx.Err()` guard after the walk. A
|
||||||
cancelled walk yields a partial size census, and every file it never reached
|
cancelled walk yields a partial size census, and every file it never reached
|
||||||
looks vanished to the update phase. That phase's own `BeginTx` also fails on
|
looks vanished to the update phase. That phase's own `BeginTx` also fails on
|
||||||
the cancelled context before deleting anything. The guard is still the barrier
|
the cancelled context before deleting anything, but the guard is the barrier
|
||||||
once an interrupted scan may commit what it has. Tests drive `run(scan)`
|
that still holds once an interrupted scan may commit what it has. Tests drive
|
||||||
against a database whose insert trigger aborts, and assert that the scan fails
|
`run(scan)` against a database whose insert trigger aborts, and assert that
|
||||||
instead of hanging and that the goroutine count returns to its pre-scan
|
the scan fails instead of hanging and that the goroutine count returns to its
|
||||||
baseline. Other tests cancel a scan part-way through the walk,
|
pre-scan baseline. Other tests cancel a scan part-way through the walk and
|
||||||
deterministically, by counting the scan's own checks of `ctx.Done()`. They
|
assert that it stops at the guard with every record intact. Direct tests of
|
||||||
assert that it stops at the guard, holding a partial size census and a
|
|
||||||
still-populated record index, with every record intact. Direct tests of
|
|
||||||
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
|
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
|
||||||
and `hashPhase` cover the remaining cancellation branches of both pools
|
and `hashPhase` cover the remaining cancellation branches of both pools
|
||||||
(2026-08-09, branch `hash-pool-cleanup`,
|
(2026-08-09, branch `hash-pool-cleanup`,
|
||||||
@@ -484,13 +469,13 @@ are known.
|
|||||||
are gone, so the deferred `db.Close()`, and with it the SQLite WAL checkpoint,
|
are gone, so the deferred `db.Close()`, and with it the SQLite WAL checkpoint,
|
||||||
runs when a subcommand fails. `runScan`, `runReport`, `runTrees`,
|
runs when a subcommand fails. `runScan`, `runReport`, `runTrees`,
|
||||||
`loadRecords` and `resolveRoots` return errors instead. The single exit point
|
`loadRecords` and `resolveRoots` return errors instead. The single exit point
|
||||||
is `run` in `main.go`. It maps a `fatalError`, anything a subcommand returned,
|
is `run` in `main.go`. It maps a `fatalError` to exit 1 and cobra's own
|
||||||
to exit 1, and cobra's own argument and flag errors to exit 2. So a runtime
|
argument and flag errors to exit 2, so a runtime failure is never reported as
|
||||||
failure is never reported as a usage error, nor printed with the usage text.
|
a usage error or printed with the usage text. `main_test.go` drives the CLI
|
||||||
`main_test.go` drives the CLI in-process and asserts the exit codes from
|
in-process and asserts the exit codes from README "Error handling" and the
|
||||||
README "Error handling" and the stdout/stderr split. That includes a fatal
|
stdout/stderr split. That includes a fatal error after the database is open
|
||||||
error after the database is open leaving no `-wal` or `-shm` file behind, for
|
leaving no `-wal` or `-shm` file behind, for `scan`, `report` or `trees`
|
||||||
`scan`, `report` or `trees` (2026-08-09, branch `db-close-on-fatal`,
|
(2026-08-09, branch `db-close-on-fatal`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||||
|
|
||||||
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as
|
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as
|
||||||
@@ -501,17 +486,17 @@ are known.
|
|||||||
schema, so they are applied. No new lint findings surfaced (2026-08-09, branch
|
schema, so they are applied. No new lint findings surfaced (2026-08-09, branch
|
||||||
`golangci-v2.12.2`, https://git.eeqj.de/sneak/sfdupes/issues/3)
|
`golangci-v2.12.2`, https://git.eeqj.de/sneak/sfdupes/issues/3)
|
||||||
|
|
||||||
- `Makefile` targets converted to scripts-to-rule-them-all `script/`
|
- Makefile targets converted to scripts-to-rule-them-all `script/` entrypoints,
|
||||||
entrypoints, like the other managed repos. All 12 entrypoints exist:
|
like the other managed repos. All 12 entrypoints exist:
|
||||||
- `bootstrap`, `setup` and `projectname`;
|
- `bootstrap`, `setup` and `projectname`;
|
||||||
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
|
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
|
||||||
- `docker`, `cibuild`, `precommit` and `install-precommit`.
|
- `docker`, `cibuild`, `precommit` and `install-precommit`.
|
||||||
|
|
||||||
Every `Makefile` target does nothing but call its script (2026-07-26, commit
|
Every Makefile target is a thin shim over them (2026-07-26, commit
|
||||||
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
|
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
|
||||||
|
|
||||||
- The binary is the default `make` target: plain `make` builds `sfdupes`, where
|
- The binary is the default Make target: plain `make` builds `sfdupes`, where it
|
||||||
it used to run `check` plus `build`. `make build` remains as an alias
|
used to run `check` plus `build`. `make build` remains as an alias
|
||||||
(2026-07-24, branch `make-default-target`)
|
(2026-07-24, branch `make-default-target`)
|
||||||
|
|
||||||
- Scan-wide phases, concurrent operands and batched updates. All operands seed
|
- Scan-wide phases, concurrent operands and batched updates. All operands seed
|
||||||
@@ -573,11 +558,9 @@ are known.
|
|||||||
- `LICENSE` and the README License section. MIT was assumed from house
|
- `LICENSE` and the README License section. MIT was assumed from house
|
||||||
convention, for the owner to confirm;
|
convention, for the owner to confirm;
|
||||||
- `REPO_POLICIES.md` in the repo root;
|
- `REPO_POLICIES.md` in the repo root;
|
||||||
- the canonical `.golangci.yml`, with the 150 findings it raised fixed, so
|
- the canonical `.golangci.yml`, with the 150 findings it raised fixed;
|
||||||
that `make lint` is clean;
|
|
||||||
- the `Makefile` targets `test`, `lint`, `fmt`, `fmt-check`, `docker` and
|
- the `Makefile` targets `test`, `lint`, `fmt`, `fmt-check`, `docker` and
|
||||||
`hooks`. `check` depended on `build`, which wrote the binary, and
|
`hooks`, and a `check` target that no longer depends on `build`;
|
||||||
`make check` must not modify files, so `check` no longer depends on it;
|
|
||||||
- a test suite, which the policy required with a 30-second timeout and the
|
- a test suite, which the policy required with a 30-second timeout and the
|
||||||
conditional `-v` rerun. It covers parsing, grouping, digests, suppression,
|
conditional `-v` rerun. It covers parsing, grouping, digests, suppression,
|
||||||
hashing and the scan pipeline, at 64% coverage;
|
hashing and the scan pipeline, at 64% coverage;
|
||||||
@@ -589,7 +572,7 @@ are known.
|
|||||||
name, purpose, category, license and author, then Getting Started,
|
name, purpose, category, license and author, then Getting Started,
|
||||||
Rationale, TODO, License and Author;
|
Rationale, TODO, License and Author;
|
||||||
- the stale README non-goal "no git repository setup and no CI" removed;
|
- the stale README non-goal "no git repository setup and no CI" removed;
|
||||||
- the pre-commit hook, installed with `make hooks` once that target existed.
|
- the pre-commit hook installed.
|
||||||
|
|
||||||
The code was already `gofmt`-clean, so no standalone formatting commit was
|
The code was already `gofmt`-clean, so no standalone formatting commit was
|
||||||
needed (2026-07-23, branch `repo-policy-compliance`)
|
needed (2026-07-23, branch `repo-policy-compliance`)
|
||||||
|
|||||||
Reference in new issue
Block a user