Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
16a57547b8 |
@@ -1,35 +1,47 @@
|
||||
# Workflow
|
||||
|
||||
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
|
||||
tracker gets filed as an issue first
|
||||
- branch from `next`
|
||||
- do the work, with tests, in small focused commits
|
||||
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
|
||||
as the work)
|
||||
- push the branch and open a PR against `next` whose title ends with
|
||||
` (closes #N)`
|
||||
- an independent review gates each merge to `next`; every finding is addressed
|
||||
or explicitly rebutted on the PR
|
||||
- only the owner merges `next` to `main`
|
||||
- Take an issue from the `1.0.0` milestone on the tracker. Work that is not on
|
||||
the tracker yet is filed as an issue first.
|
||||
- Branch from `next`.
|
||||
- Do the work, with tests, in small focused commits.
|
||||
- Record the step at the top of Completed Steps, in the same commit as the work.
|
||||
- Push the branch and open a PR against `next`. Its title ends with
|
||||
` (closes #N)`.
|
||||
- An independent review gates each merge to `next`. Every finding is addressed
|
||||
or rebutted on the PR.
|
||||
- Only the owner merges `next` into `main`.
|
||||
|
||||
# Status
|
||||
|
||||
- pre-1.0
|
||||
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
|
||||
under the `1.0.0` milestone are what remains before the tag, and this file
|
||||
records history and process, not the queue
|
||||
- Pre-1.0.
|
||||
- The tracker is authoritative for the pre-1.0 backlog. The open issues under
|
||||
the `1.0.0` milestone are what remains before the tag. This file records
|
||||
history and process, not the queue.
|
||||
|
||||
# Next Step
|
||||
|
||||
- take the next issue from the `1.0.0` milestone on the tracker:
|
||||
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
|
||||
of truth for what is left before 1.0.0. Individual issues are deliberately not
|
||||
restated here; a copy in this file drifts out of date the moment the tracker
|
||||
moves
|
||||
- Take the next issue from the `1.0.0` milestone:
|
||||
https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated
|
||||
here, because a copy drifts as soon as the tracker moves.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- a file or directory removed while the scan runs is not an error, in the walk
|
||||
Newest first. An entry ends with its date and its issue or branch, where those
|
||||
are known.
|
||||
|
||||
- `TODO.md` rewritten from scratch by the same rules as `README.md`: short
|
||||
sentences, lists as bullets, nothing a newcomer reads twice. Every completed
|
||||
step is kept, with its date and its issue or branch. The policy-compliance
|
||||
checklist of 2026-07-22 is folded into its step (2026-10-26,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
|
||||
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
|
||||
it. It leads with what the program is and how to use it. A reference section
|
||||
covers the settings, the database, the output formats and the exit codes;
|
||||
development and testing come last. Every statement was checked against the
|
||||
code on `next` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
|
||||
- A file or directory removed while the scan runs is not an error, in the walk
|
||||
or in the hash phase. The scan prints no line for it and does not count it as
|
||||
skipped. When the first of several hard links to a file is gone, the hash
|
||||
phase reads the next one instead. On a terminal every progress display starts
|
||||
@@ -39,218 +51,236 @@
|
||||
terminal now loses its rate before its bar. The time remaining is labelled
|
||||
`remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133)
|
||||
|
||||
- the `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
|
||||
- The `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
|
||||
bytes of a SHA-256, half the size of the hex text they held. The hashes stay
|
||||
raw from the hash phase through the database to `report` and `trees`. Reading
|
||||
a record whose hash is not 32 bytes is an error (2026-10-26,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/134)
|
||||
|
||||
- the hash phase gives every file the walk records three hashes: `head` and
|
||||
`tail`, the SHA-256 of its first and last 64 KiB, and `content`, the SHA-256
|
||||
of the whole file under 100 MiB and of 1 MiB at each gigabyte from 100 MiB up.
|
||||
The 10 MiB and 50 MiB thresholds, the size census, the records stored without
|
||||
hashes and the content phase are gone, so a scan has four phases and a rescan
|
||||
reads only new and changed files. `info` no longer counts the records with
|
||||
hashes, since every record has them (2026-10-25,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/132)
|
||||
- Every file the walk records gets three hashes in the hash phase. `head` and
|
||||
`tail` are the SHA-256 of its first and last 64 KiB. `content` is the SHA-256
|
||||
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
|
||||
up. Gone are:
|
||||
- the 10 MiB and 50 MiB thresholds;
|
||||
- the size census;
|
||||
- the records without hashes;
|
||||
- the content phase.
|
||||
|
||||
- re-vendor `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` from
|
||||
`sneak/prompts` `next` at `0653579`: in the `lint`, `test` and `builder`
|
||||
stages of the `Dockerfile` the module download, and in `builder` the
|
||||
system-package install, now come before `ARG CACHEBUST`, so Docker keeps them
|
||||
until the base image, the install line, `go.mod` or `go.sum` changes, and
|
||||
every step that lints, tests or compiles still comes after it;
|
||||
`golangci-lint run` also keeps golangci-lint's cache. README says the same
|
||||
A scan has four phases, and a rescan reads only new and changed files.
|
||||
`info` no longer counts the records with `head` and `tail` hashes, nor those
|
||||
with a content hash, since every record has all three (2026-10-25,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/132)
|
||||
|
||||
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` re-vendored from
|
||||
`sneak/prompts` `next` at `0653579`. Two steps now come before
|
||||
`ARG CACHEBUST`: the module download in the `lint`, `test` and `builder`
|
||||
stages of the `Dockerfile`, and the package install in `builder`. Docker keeps
|
||||
them until the base image, the install line, `go.mod` or `go.sum` changes.
|
||||
Every step that lints, tests or compiles still comes after it.
|
||||
`golangci-lint run` also keeps the linter's own cache. README says the same
|
||||
(2026-10-18, https://git.eeqj.de/sneak/sfdupes/issues/109)
|
||||
|
||||
- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so
|
||||
`go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has
|
||||
passed the build; README "Install" says so, says what `@main` and `v0.0.1`
|
||||
give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the
|
||||
early build without a database. When the linker set no version,
|
||||
`sfdupes --version` prints the main module's version that Go recorded in the
|
||||
binary, the tag or the pseudo-version of a `go install` build, and `dev` only
|
||||
when neither is known (2026-10-17,
|
||||
- Release rule set: until 1.0, each tip of `main` that passes the build gets the
|
||||
next `v0.0.N` tag, so `go install sneak.berlin/go/sfdupes@latest` installs it.
|
||||
`go.mod` retracts `v0.0.1`, the early build without a database. When the
|
||||
linker set no version, `sfdupes --version` prints the version Go recorded in
|
||||
the binary: the tag, or the pseudo-version of a `go install` build. It prints
|
||||
`dev` only when neither is known. README "Install" documented the rule, what
|
||||
`@main` and `v0.0.1` give, and how to check an installed binary (2026-10-17,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/116)
|
||||
|
||||
- `scan` creates the database's directory where the database path leads: the
|
||||
path with its last element cut off as text, never cleaned, so a `..` after a
|
||||
symlinked directory is followed through the symlink, as when the file is
|
||||
opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
|
||||
symlink to `/data/x`, it creates `/data/new` and the database in it, where it
|
||||
used to create `/srv/new` and then fail (2026-10-17,
|
||||
- `scan` creates the database's directory where the database path leads. The
|
||||
directory is the path with its last element cut off as text, never cleaned. A
|
||||
`..` after a symlinked directory therefore goes through the symlink, as it
|
||||
does when the file is opened. With
|
||||
`SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a symlink to
|
||||
`/data/x`, it now creates `/data/new` and the database in it. It used to
|
||||
create `/srv/new` and then fail (2026-10-17,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/119)
|
||||
|
||||
- `scan` never records its own database: once the database and its `.lock` file
|
||||
- `scan` never records its own database. Once the database and its `.lock` file
|
||||
are open, it notes the device and inode of the database file, its `-wal` and
|
||||
`-shm` files and its `.lock` file, and the walk passes over a file with one of
|
||||
them wherever the operands reach it, through a symlinked directory included,
|
||||
printing no line and counting nothing. A record an earlier scan left for one
|
||||
of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
|
||||
`-shm` files and the `.lock` file. The walk passes over a file with one of
|
||||
them wherever it reaches it, through a symlinked directory included, printing
|
||||
no line and counting nothing. A record an earlier scan left for one of them is
|
||||
deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
|
||||
|
||||
- `scan` never records or reads anything that is not on a persistent filesystem
|
||||
(`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
|
||||
matched by the device ID of each such filesystem, listed as the scan starts
|
||||
from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
|
||||
however the walk or the content phase reaches it; no flag turns that off.
|
||||
Below an operand, its walk does not enter the default excludes either, so
|
||||
`sfdupes scan /` skips the kernel's view of the running system and the
|
||||
temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
|
||||
`/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
|
||||
`/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
|
||||
Windows list, not used yet. An operand at or inside a default exclude is still
|
||||
- `scan` never records or reads anything on a filesystem that is not persistent:
|
||||
`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types. It
|
||||
matches them by device ID. As the scan starts, it lists the mounted
|
||||
filesystems of those types, from `/proc/self/mountinfo` on Linux and from the
|
||||
mount list on macOS. No flag turns that off. Below an operand, the walk does
|
||||
not enter the default excludes either, so `sfdupes scan /` skips the kernel's
|
||||
view of the running system and the temporary directories. On Linux those are
|
||||
`/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`. On macOS they are
|
||||
`/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
|
||||
`/System/Volumes/VM` and `/System/Volumes/Data`. A Windows list exists in the
|
||||
code but is not used. An operand at or inside a default exclude is still
|
||||
walked. The scan prints a line for each path it leaves out, without counting
|
||||
it as skipped, and the records there count as outside the scanned operands, as
|
||||
for `-x`. `--no-default-excludes` turns off the default excludes only, and
|
||||
`scan --help` lists them (2026-10-17,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
||||
|
||||
- the rate a progress display shows covers the items completed over the last 5
|
||||
seconds, or since the line before when stderr is not a terminal, so it falls
|
||||
to 0 while nothing completes, and a burst after a wait no longer reads as a
|
||||
steady rate of over 1,000 files a second; the ETA still comes from the phase's
|
||||
average. Once no item has completed for 10 seconds, the word `stalled` and
|
||||
where the phase waits take the rate's place, as `stalled disk/db`: `disk`,
|
||||
`db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
|
||||
either, each phase giving its word where it is defined. A line too wide for
|
||||
the terminal loses its bar first, then its rate or those words, so the phase,
|
||||
progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts
|
||||
like those of README's example, the content phase's checks and reads lose
|
||||
those words at most times in minutes, and every other phase keeps them
|
||||
whenever the elapsed time and the ETA are both in minutes. The content phase's
|
||||
last partial batch is now written under a display of its own (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||
- The rate a progress display shows covers the items completed over the last 5
|
||||
seconds, or since the line before when stderr is not a terminal. It falls to 0
|
||||
while nothing completes, and a burst after a wait no longer reads as a steady
|
||||
rate of over 1,000 files a second. The time remaining still comes from the
|
||||
phase's average. Once no item has completed for 10 seconds, the word `stalled`
|
||||
and where the phase waits take the rate's place, as `stalled disk/db`. The
|
||||
places are `disk`, `db`, `out` for stdout and `cpu`, or two of them for a
|
||||
phase that may wait on either. Each phase gives its word where it is defined.
|
||||
A line too wide for the terminal loses its bar first, then its rate or those
|
||||
words, so the phase, progress, elapsed time and ETA stay whole. On an
|
||||
80-column terminal, at counts like those of README's example, the content
|
||||
phase's checks and reads lose those words at most times in minutes. Every
|
||||
other phase keeps them whenever the elapsed time and the ETA are both in
|
||||
minutes. The content phase's last partial batch is written under a display of
|
||||
its own (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||
|
||||
- new subcommand `info` prints the database's details on stdout, one per line as
|
||||
a name, a tab and the value: its path and where it came from, the sizes of the
|
||||
database file and of its `-wal` file when there is one, when the database file
|
||||
was last changed, the schema version, the number of records, how many have
|
||||
head and tail hashes and how many a content hash, and the sizes of all records
|
||||
added up. It opens the database as `report` does, and shows two phases on
|
||||
stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come
|
||||
from one read-only transaction, the counts from one query (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/117)
|
||||
- New subcommand `info` prints the database's details on stdout, one per line as
|
||||
a name, a tab and the value:
|
||||
- the path and where it came from;
|
||||
- the sizes of the database file and of its `-wal` file, when there is one;
|
||||
- when the database file was last changed;
|
||||
- the schema version;
|
||||
- the number of records, how many have `head` and `tail` hashes, and how
|
||||
many have a content hash;
|
||||
- the sizes of all records added up.
|
||||
|
||||
It opens the database as `report` does and shows two phases on stderr,
|
||||
`[1/2] open` and `[2/2] count`. The schema version and the counts come from
|
||||
one read-only transaction, the counts from one query (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/117)
|
||||
|
||||
- `report` shows its work on stderr in two phases, `[1/2] open` and
|
||||
`[2/2] read`, with the progress display `scan` and `trees` use, from the
|
||||
moment it starts until the summary line: `open` says it is opening the
|
||||
database, also while it waits on a busy one; `read` counts the records, then
|
||||
reads them against that count in signature order, which SQLite takes from the
|
||||
`files_signature` index without sorting, so the first record comes at once.
|
||||
`report` forms the duplicate groups itself, holding only those of the size it
|
||||
is reading, and writes them ordered by first path before it reads the next
|
||||
size. stdout and the summary line are unchanged, and SQLite no longer needs a
|
||||
temporary file to sort in (2026-10-16,
|
||||
`[2/2] read`, with the progress display `scan` and `trees` use. The display
|
||||
runs from the moment `report` starts until the summary line. `open` says it is
|
||||
opening the database, also while it waits on a busy one. `read` counts the
|
||||
records, then reads them against that count in signature order. SQLite takes
|
||||
that order from the `files_signature` index without sorting, so the first
|
||||
record comes at once. `report` forms the duplicate groups itself, holding only
|
||||
those of the size it is reading, and writes them ordered by first path before
|
||||
it reads the next size. stdout and the summary line are unchanged, and SQLite
|
||||
no longer needs a temporary file to sort in (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/114)
|
||||
|
||||
- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
|
||||
`[3/4] group` and `[4/4] write`, each with the progress display `scan` uses,
|
||||
from the moment it starts until the summary line: `open` says it is opening
|
||||
the database, also while it waits on a busy one; `read` counts the records,
|
||||
then reads them against that count; `group` counts the directories it groups,
|
||||
and `write` the groups it writes. A display's rate names what its phase
|
||||
counts: files, records, directories or groups. stdout and the summary line are
|
||||
unchanged. When stdout and stderr are the same file, as one terminal is, none
|
||||
of stdout's lines shares a line with the display; when they are not, the
|
||||
display keeps being drawn while stdout waits on a slow reader (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/113)
|
||||
`[3/4] group` and `[4/4] write`, each with the progress display `scan` uses.
|
||||
The display runs from the moment `trees` starts until the summary line. `open`
|
||||
says it is opening the database, also while it waits on a busy one. `read`
|
||||
counts the records, then reads them against that count. `group` counts the
|
||||
directories it groups, and `write` the groups it writes. A display's rate
|
||||
names what its phase counts: files, records, directories or groups. stdout and
|
||||
the summary line are unchanged. When stdout and stderr are the same file, as
|
||||
one terminal is, no line of stdout shares a line with the display. When they
|
||||
are not, the display keeps being drawn while stdout waits on a slow reader
|
||||
(2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113)
|
||||
|
||||
- each `scan` phase's progress display names the phase and its place among the
|
||||
five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed
|
||||
time and ETA: `load` counts the records first, the walk's total is estimated
|
||||
from the records under the operands, and the content phase counts the records
|
||||
it checks from SQLite's grouping alone, then says it is searching the database
|
||||
until the first one is returned. On a terminal a phase with an exact total
|
||||
draws its line followed by the library's bar, and any other a spinner followed
|
||||
by the line; every display is drawn again several times a second. No line
|
||||
reaches the terminal's last column, however narrow, also after the terminal is
|
||||
made narrower during a phase: a line too wide leaves out its rate, then its
|
||||
bar, keeping the phase, progress, elapsed time and ETA whole, and is cut short
|
||||
only when those alone do not fit, as below about 70 columns late in a large
|
||||
scan. In plain mode the line is printed every 5 seconds whether or not an item
|
||||
completed (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||
- Each `scan` phase's progress display names the phase and its place among the
|
||||
five, as `[2/5] walk`. It runs from the moment the phase starts and shows its
|
||||
progress, elapsed time and ETA. `load` counts the records first. The walk's
|
||||
total is estimated from the records under the operands. The content phase
|
||||
counts the records it checks from SQLite's grouping alone, then says it is
|
||||
searching the database until the first one is returned. On a terminal a phase
|
||||
with an exact total draws its line followed by the library's bar. Any other
|
||||
phase draws a spinner followed by the line. Every display is drawn again
|
||||
several times a second. No line reaches the terminal's last column, however
|
||||
narrow, also after the terminal is made narrower during a phase. A line too
|
||||
wide leaves out its rate, then its bar, keeping the phase, progress, elapsed
|
||||
time and ETA whole. It is cut short only when those alone do not fit, as below
|
||||
about 70 columns late in a large scan. In plain mode the line is printed every
|
||||
5 seconds whether or not an item completed (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||
|
||||
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
||||
they use, as an absolute path, and whether that path is the default or comes
|
||||
from `SFDUPES_DATABASE`; `scan` says when the database is new. README
|
||||
from `SFDUPES_DATABASE`. `scan` says when the database is new. README
|
||||
"Database" says when `scan` writes each file (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/111)
|
||||
|
||||
- re-vendor the canonical files and model scripts from `sneak/prompts` `next` at
|
||||
`e3f79c4`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile`
|
||||
that write no image, and `make test` runs the suite under the race detector,
|
||||
so `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race` are
|
||||
gone; every `docker build` in `script/` passes a new `CACHEBUST` value, which
|
||||
every stage of the `Dockerfile` declares first, so every `RUN` step runs on
|
||||
every build, while the steps that compile keep Go's build cache and the tests
|
||||
run with `-count=1`; prettier runs on the host, from the node and yarn
|
||||
`script/bootstrap` installs, so the `prettier` and `markdown` stages are gone
|
||||
and the build stage installs `git` and `make` itself; a new push cancels the
|
||||
workflow's older run on the same branch, and a run stops after 20 minutes;
|
||||
`.claude/settings.json` is deleted (2026-10-15,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95). Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/109 the module download in `lint`,
|
||||
`test` and the build stage, and the install of `git` and `make` in the build
|
||||
stage, come before `ARG CACHEBUST`, and Docker keeps them between builds;
|
||||
every step that lints, tests or compiles still runs on every build
|
||||
- The canonical files and model scripts re-vendored from `sneak/prompts` `next`
|
||||
at `e3f79c4`:
|
||||
- `golangci-lint` v2.14.0;
|
||||
- lint and test are phases of the `Dockerfile` that write no image, and
|
||||
`make test` runs the suite under the race detector. `Dockerfile.lint`,
|
||||
`script/verify-lint-image-pin` and `make test-race` are gone;
|
||||
- every `docker build` in `script/` passes a new `CACHEBUST` value, which
|
||||
every stage of the `Dockerfile` declares first, so every `RUN` step runs
|
||||
on every build. The steps that compile keep Go's build cache, and the
|
||||
tests run with `-count=1`;
|
||||
- `prettier` runs on the host, from the `node` and `yarn` that
|
||||
`script/bootstrap` installs, so the `prettier` and `markdown` stages are
|
||||
gone and the build stage installs `git` and `make` itself;
|
||||
- a new push cancels the workflow's older run on the same branch, and a run
|
||||
stops after 20 minutes;
|
||||
- `.claude/settings.json` is deleted.
|
||||
|
||||
- with `-x`, an operand under another operand is walked on its own instead of
|
||||
being dropped when a directory from it (from its directory, for a regular
|
||||
file) up to but not including that operand is on a different filesystem than
|
||||
that operand, whose walk then cannot reach it, and the records of files gone
|
||||
from below it are deleted (2026-10-14,
|
||||
Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before
|
||||
`ARG CACHEBUST`: the module download in `lint`, `test` and the build stage,
|
||||
and the install of `git` and `make` in the build stage. Docker keeps them
|
||||
between builds, and every step that lints, tests or compiles still runs on
|
||||
every build (2026-10-15, https://git.eeqj.de/sneak/sfdupes/issues/95)
|
||||
|
||||
- With `-x`, an operand under another operand is no longer dropped when the
|
||||
outer operand's walk cannot reach it. That is the case when a directory from
|
||||
the inner operand up to, but not including, the outer one is on a different
|
||||
filesystem than the outer one. The inner operand is then walked on its own.
|
||||
For a regular-file operand the directories checked start at its directory. The
|
||||
records of files gone from below it are deleted (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/106)
|
||||
|
||||
- with `-x`, `scan` keeps the records below a directory on another filesystem
|
||||
- With `-x`, `scan` keeps the records below a directory on another filesystem
|
||||
under its operand, and counts their sizes in the size census, instead of
|
||||
deleting them (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
|
||||
|
||||
- the hash and content phases open each file without blocking and check that it
|
||||
is still a regular file, so one replaced by a FIFO or device node after it was
|
||||
- The hash and content phases open each file without blocking and check that it
|
||||
is still a regular file. A file replaced by a FIFO or device node after it was
|
||||
statted is warned about and skipped instead of hanging the scan (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/100)
|
||||
|
||||
- the scan lock file is named from the database's path with symlinks resolved,
|
||||
after `scan` creates the database file if it is missing, so a scan through a
|
||||
symlink to the database fails while another scan holds the lock (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/101)
|
||||
- The scan lock file is named from the database's path with symlinks resolved,
|
||||
after `scan` creates the database file if it is missing. A scan through a
|
||||
symlink to the database therefore fails while another scan holds the lock
|
||||
(2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/101)
|
||||
|
||||
- on a terminal, a warning for the last file of a phase no longer erases the
|
||||
phase's finished bar: the bar is drawn again below the warning (2026-10-14,
|
||||
- On a terminal, a warning for the last file of a phase no longer erases the
|
||||
phase's finished bar. The bar is drawn again below the warning (2026-10-14,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/102)
|
||||
|
||||
- `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus
|
||||
`mtime_nsec`, and compares it at that resolution, so a same-size rewrite
|
||||
within the same second is re-hashed (2026-10-07,
|
||||
- `scan` records the mtime to the nanosecond, as whole seconds in `mtime` plus
|
||||
`mtime_nsec`, and compares it at that resolution. A same-size rewrite within
|
||||
the same second is hashed again (2026-10-07,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/12)
|
||||
|
||||
- cut the narration from `TODO.md` Completed Steps and from the comments in
|
||||
- The narration was cut from `TODO.md` Completed Steps and from the comments in
|
||||
`script/`, `Dockerfile` and `Dockerfile.lint` (gone since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95); §Workflow now branches from and
|
||||
merges to `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95). The Workflow section now
|
||||
branches from and merges to `next` (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/49)
|
||||
|
||||
- `make test-race` ran the test suite under the race detector in a cgo-enabled
|
||||
container, outside `make check` (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/18). Since
|
||||
container, outside `make check`. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
|
||||
under the race detector, in the `Dockerfile`'s Debian-based `test` phase, and
|
||||
`make test-race` is gone
|
||||
`make test-race` is gone (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/18)
|
||||
|
||||
- a bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
|
||||
rather than serve the gates from cache (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/39). Since
|
||||
- A bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
|
||||
rather than serve the gates from cache. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
|
||||
`REPO_POLICIES.md` requires, and may serve the gates from cache; the builds in
|
||||
`script/` pass a new `CACHEBUST` value, so theirs always run
|
||||
`REPO_POLICIES.md` requires, and may serve the gates from cache. The builds in
|
||||
`script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/39)
|
||||
|
||||
- `make fmt` and `make fmt-check` run prettier over all Markdown, and CI checks
|
||||
it; all Markdown reformatted (2026-10-04,
|
||||
- `make fmt` and `make fmt-check` run `prettier` over all Markdown, and CI
|
||||
checks it. All Markdown was reformatted (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/19)
|
||||
|
||||
- `script/lint` writes no image, so a run no longer leaves an untagged one
|
||||
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
|
||||
|
||||
- tests cover a missing database, `scan` keeping stdout empty, its skip warning,
|
||||
- Tests cover a missing database, `scan` keeping stdout empty, its skip warning,
|
||||
the `report` and `trees` summary lines, and every subcommand going through
|
||||
`runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
||||
|
||||
@@ -258,296 +288,307 @@
|
||||
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/26)
|
||||
|
||||
- a test fails when either `hashWorker` cancellation check in `scan.go` is
|
||||
- A test fails when either `hashWorker` cancellation check in `scan.go` is
|
||||
removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)
|
||||
|
||||
- a database path holding `?`, `#` or `%` opens exactly the file it names
|
||||
- A database path holding `?`, `#` or `%` opens exactly the file it names
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
|
||||
|
||||
- `scan` rejects `--workers` below 1 as a usage error instead of running
|
||||
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
|
||||
|
||||
- a test fails when either walk cancellation check in `scan.go` is removed
|
||||
- A test fails when either walk cancellation check in `scan.go` is removed
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
|
||||
|
||||
- test that `scan` refuses a database with another schema version (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/64)
|
||||
- A test checks that `scan` refuses a database with another schema version
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
|
||||
|
||||
- correct four inaccurate comments in `cancel_test.go` and rename
|
||||
`walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04,
|
||||
- Four inaccurate comments in `cancel_test.go` corrected, and
|
||||
`walkCancelInFlightDirs` renamed to `walkCancelInFlightFiles` (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/33)
|
||||
|
||||
- test the `-x` filesystem-boundary rules in `subdirJob` (2026-10-04,
|
||||
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/17)
|
||||
|
||||
- `scan` creates the schema in one transaction; a version-0 database with a
|
||||
- `scan` creates the schema in one transaction. A version-0 database with a
|
||||
`files` table is refused with a clear schema-version error (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/11)
|
||||
|
||||
- README documents install, Docker, a daily cron scan and how to read and check
|
||||
the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
|
||||
|
||||
- the `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
||||
and copied the sources with `--chown`, so no `chown -R` walked them
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43). Since
|
||||
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
||||
and copied the sources with `--chown`, so no `chown -R` walked them. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
|
||||
nothing changes owner: the build stage only compiles, as root, and the tests
|
||||
run as `nobody` in the `test` phase
|
||||
run as `nobody` in the `test` phase (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/43)
|
||||
|
||||
- `--version` prints `sfdupes VERSION` to stdout; README documents it and
|
||||
- `--version` prints `sfdupes VERSION` to stdout. README documents it and
|
||||
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
|
||||
|
||||
- `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has hashed,
|
||||
deletes nothing more, exits 1 (2026-10-04,
|
||||
- `scan` stops cleanly on `SIGINT` or `SIGTERM`: it commits what it has hashed,
|
||||
deletes nothing more, and exits 1 (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/5)
|
||||
|
||||
- `report` and `trees` stream the records instead of holding them all in memory;
|
||||
the schema gains the `files_signature` index (2026-10-04,
|
||||
- `report` and `trees` stream the records instead of holding them all in memory.
|
||||
The schema gains the `files_signature` index (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/14)
|
||||
|
||||
- progress prints at once on a non-terminal, uses a real terminal test, and
|
||||
- Progress prints at once on a non-terminal, uses a real terminal test, and
|
||||
prints warnings through a spinner instead of racing its redraw (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/13)
|
||||
|
||||
- warn about and skip symlink, socket, FIFO, device and `.zfs` operands, keeping
|
||||
the records beneath them (2026-10-03,
|
||||
- Symlink, socket, FIFO, device and `.zfs` operands are warned about and
|
||||
skipped, and the records beneath them are kept (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/9)
|
||||
|
||||
- `scan` holds a lock on a lock file beside the database for its whole run, so a
|
||||
second `scan` fails at once with exit 1 (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/53)
|
||||
|
||||
- test stdout write failures in `report` and `trees`; README states that
|
||||
`| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null`
|
||||
- Tests cover stdout write failures in `report` and `trees`. README states that
|
||||
`| head` ends `sfdupes` by `SIGPIPE` and that `>&-` writes to `/dev/null`
|
||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)
|
||||
|
||||
- `report` and `trees` open the database read-only, and `scan` leaves it out of
|
||||
WAL mode, so reading needs only read access (2026-10-03, closes
|
||||
WAL mode, so reading needs only read access (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/8)
|
||||
|
||||
- escape tabs, newlines, carriage returns and backslashes in report, trees and
|
||||
warning paths; the root directory's path is `/` (2026-10-03,
|
||||
- Tabs, newlines, carriage returns and backslashes are escaped in report, trees
|
||||
and warning paths. The root directory's path is `/` (2026-10-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
||||
|
||||
- stamp the git tag or short commit in a plain `docker build .` instead of `dev`
|
||||
(2026-10-02, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git`
|
||||
without `.git/config`; the build stage stamps the `VERSION` build argument,
|
||||
else `git describe --tags --always`, and fails if the context carries `.git`
|
||||
and the version is still empty, `dev` or `unknown`. CI checks out the full
|
||||
history (`fetch-depth: 0`) so it stamps the same value as `make build`.
|
||||
- A plain `docker build .` stamps the git tag or short commit instead of `dev`.
|
||||
`.dockerignore` sends `.git` without `.git/config`. The build stage stamps the
|
||||
`VERSION` build argument, else `git describe --tags --always`. It fails if the
|
||||
context carries `.git` and the version is still empty, `dev` or `unknown`. CI
|
||||
checks out the full history (`fetch-depth: 0`), so it stamps the same value as
|
||||
`make build` (2026-10-02, branch `next`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/67)
|
||||
|
||||
- replace the 1 KiB end-window sampling with the head/tail plus content-hash
|
||||
ladder (2026-09-22, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection"
|
||||
documents every rung. A file under 10 MiB is hashed in full, and its `head`,
|
||||
`tail` and `content` all hold that hash. A larger file gets only its 64 KiB
|
||||
`head` and `tail` in the hash phase; the content phase, after the update
|
||||
phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1
|
||||
MiB samples at or above) only when its size, `head` and `tail` match another
|
||||
record's from this scan or an earlier one, and never reads a file gone or
|
||||
changed since its record was written. `report` and `trees` leave out any
|
||||
record without a `content` hash. The `content` column is part of the version 1
|
||||
schema.
|
||||
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
|
||||
size, and README "Duplicate detection" documented every size threshold. A file
|
||||
under 10 MiB was hashed in full, and its `head`, `tail` and `content` all held
|
||||
that hash. A larger file got only its 64 KiB `head` and `tail` in the hash
|
||||
phase. The content phase, after the update phase, read it for `content` only
|
||||
when its size, `head` and `tail` matched another record's, from this scan or
|
||||
an earlier one. `content` was the whole file below 50 MiB, and gigabyte-spaced
|
||||
1 MiB samples at or above. A file gone or changed since its record was written
|
||||
was never read. `report` and `trees` left out any record without a `content`
|
||||
hash. The `content` column is part of the version 1 schema. Superseded by
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/132 (2026-09-22, branch `next`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/61)
|
||||
|
||||
- remove the dead `files.dat` references from `Makefile`, `.gitignore` and
|
||||
`.dockerignore` (2026-09-21, branch `next`, closes
|
||||
- The dead `files.dat` references removed from `Makefile`, `.gitignore` and
|
||||
`.dockerignore` (2026-09-21, branch `next`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/22)
|
||||
|
||||
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
|
||||
`Dockerfile.lint` (2026-08-10, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/25): both pins became the policy
|
||||
- The lint-image pin comments and `FROM` form in `Dockerfile` and
|
||||
`Dockerfile.lint` fixed. Both pins became the policy
|
||||
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
|
||||
without the false `(Debian-based)` note or the tag. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
|
||||
holds the only golangci-lint pin, in that form, so `Dockerfile.lint` and
|
||||
`script/verify-lint-image-pin`, which compared the two pins, are gone.
|
||||
holds the only `golangci-lint` pin, in that form. `Dockerfile.lint` and
|
||||
`script/verify-lint-image-pin`, which compared the two pins, are gone
|
||||
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/25)
|
||||
|
||||
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
|
||||
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
|
||||
- All linting ran in Docker, via `Dockerfile.lint` and `script/lint`. Per the
|
||||
owner ruling the linter is never installed on a host, and
|
||||
`golangci-lint config verify` runs before `golangci-lint run`.
|
||||
`script/bootstrap` stopped installing or pinning the linter, and
|
||||
`script/verify-linter-pin` was retired. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
|
||||
`Dockerfile`'s `lint` phase, which `script/lint` builds alone and the build
|
||||
stage depends on through `COPY --from=lint /src/go.sum /dev/null`;
|
||||
stage depends on through `COPY --from=lint /src/go.sum /dev/null`.
|
||||
`Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
|
||||
an image build may run docker, so the phase calls `golangci-lint` directly.
|
||||
an image build may run `docker`, so the phase calls `golangci-lint` directly
|
||||
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/46)
|
||||
|
||||
- install the Docker build stage's prerequisites by running `script/bootstrap`
|
||||
instead of `apk add --no-cache make` inline (2026-08-09, branch
|
||||
`dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42),
|
||||
with `script/verify-linter-pin` failing the build unless the linter copied
|
||||
from the lint stage was the version `script/bootstrap` pinned. Builds then
|
||||
took up to 5m14s cold, mostly in a `chown -R` of the module cache, filed as
|
||||
- The Docker build stage's prerequisites were installed by running
|
||||
`script/bootstrap` instead of `apk add --no-cache make` inline.
|
||||
`script/verify-linter-pin` failed the build unless the linter copied from the
|
||||
lint stage was the version `script/bootstrap` pinned. Builds then took up to
|
||||
5m14s cold, mostly in a `chown -R` of the module cache, filed as
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/43. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
|
||||
`make` with `apk add --no-cache` and only compiles; the `lint` and `test`
|
||||
phases are the gates
|
||||
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
|
||||
`script/docker` cannot report a green they did not earn (2026-08-09, branch
|
||||
`cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the
|
||||
`Dockerfile` copies the tree before its gates, so on an unchanged tree Docker
|
||||
served them from cache and the build exited 0 having run nothing. The fix was
|
||||
a `CHECK_EPOCH` build argument; since
|
||||
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/42)
|
||||
|
||||
- The Docker layer cache is busted for the gate steps, so `script/cibuild` and
|
||||
`script/docker` cannot report a green they did not earn. The `Dockerfile`
|
||||
copies the tree before its gates, so on an unchanged tree Docker served them
|
||||
from cache and the build exited 0 having run nothing. The fix was a
|
||||
`CHECK_EPOCH` build argument. Since
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
|
||||
passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile`
|
||||
declares before every step that lints, tests or compiles. Run as root, the
|
||||
tests fail `TestScanHardlinkRunFailsTogether`, because root reads through the
|
||||
`chmod(0)` the test relies on, so the `test` phase runs them as `nobody`
|
||||
- check the installed golangci-lint version in `script/bootstrap` instead of
|
||||
only its presence (2026-08-09, branch `bootstrap-version-check`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
|
||||
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and
|
||||
any installed version that is not the pin — older, newer, absent or
|
||||
unparseable — is reinstalled. `go install` writes into `GOBIN` (or
|
||||
`GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so
|
||||
bootstrap re-reads the effective version after installing and, on a mismatch,
|
||||
prints both paths and both versions and exits non-zero; it does not reorder
|
||||
`PATH` or delete anyone's binary. The `--version` call keeps its stderr and is
|
||||
bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep
|
||||
presence-only checks. Verified by bootstrapping this host from v2.10.1 to
|
||||
v2.12.2 and again to a no-op, and by stub runs of the script under `dash`
|
||||
covering a thirteen-input version-parse matrix, a shadowed install that must
|
||||
exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged
|
||||
binary that must hit the timeout; `make check` and `make lint` are clean at
|
||||
v2.12.2, so v2.10.1 was not hiding any findings on `main`
|
||||
- unwind the hash worker pool on the error path (2026-08-09, branch
|
||||
`hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the
|
||||
pool is now an owned, context-aware `hashPool`: every blocking send in the
|
||||
feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path
|
||||
out, and `hashPhase` defers `pool.stop()`, which cancels and then drains
|
||||
`results` until the last goroutine has exited — draining is what frees a
|
||||
worker already parked on a send. `ctx` is threaded from `cmd.Context()`
|
||||
through `runScan`, `syncScan`, both worker pools and the whole database layer,
|
||||
as the first parameter everywhere. The walk pool gets the same treatment plus
|
||||
a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size
|
||||
census, and every file it never reached looks vanished to the update phase.
|
||||
That phase's own `BeginTx` also fails on the cancelled context before deleting
|
||||
anything, but the guard is the barrier that still holds once an interrupted
|
||||
scan may commit what it has. Tests drive `run(scan)` against a database whose
|
||||
insert trigger aborts and assert that the scan fails instead of hanging and
|
||||
that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others
|
||||
cancel a scan part-way through the walk, deterministically, by counting its
|
||||
own consultations of `ctx.Done()`, and assert that it stops at the guard
|
||||
holding a partial census and a still-populated record index, with every record
|
||||
intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`,
|
||||
`feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation
|
||||
branches of both pools
|
||||
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
|
||||
`db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4):
|
||||
`fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and
|
||||
with it the SQLite WAL checkpoint — now actually runs when a subcommand fails;
|
||||
`runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return
|
||||
errors instead. The single exit point is `run` in `main.go`: it maps a
|
||||
`fatalError` (anything a subcommand returned) to exit 1 and cobra's own
|
||||
argument and flag errors to exit 2, which keeps a runtime failure from being
|
||||
reported as a usage error or printing the usage text. New `main_test.go`
|
||||
drives the CLI in-process and asserts the exit codes from README §Error
|
||||
handling plus the stdout/stderr split, including that a fatal error raised
|
||||
after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`,
|
||||
`report` or `trees`
|
||||
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
|
||||
`golangci-v2.12.2`, merged as `38a01bd`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the
|
||||
`Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and
|
||||
replaced `.golangci.yml` with the canonical file — the linter settings (`lll`,
|
||||
`funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per
|
||||
the v2 schema, so they are actually applied; no new lint findings surfaced
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
|
||||
like the other managed repos (2026-07-26, commit `3abeacf`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints
|
||||
exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
|
||||
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`)
|
||||
and every Makefile target is now a thin shim over them
|
||||
- make the binary the default Make target (2026-07-24, branch
|
||||
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran
|
||||
`check` plus `build`); `make build` remains as an alias
|
||||
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
|
||||
`scan-wide-phases`): all operands seed the shared walk pool and every pass
|
||||
runs once over the whole scan, so totals and ETAs are scan-global; the
|
||||
per-operand walk/hash/update cycles and their stderr announcements are gone;
|
||||
the update pass commits in batched transactions — the filesystem is
|
||||
authoritative and the database an eventually-consistent reflection, so
|
||||
scan-level atomicity is not required
|
||||
- split the stat pass back out of the walk (2026-07-24, branch
|
||||
`parallel-phases`): phases are strictly sequential again — walk, stat, hash,
|
||||
update per operand — with parallelism only inside each phase; the walk
|
||||
enumerates paths with per-directory workers and the stat pass lstats them with
|
||||
per-file workers, restoring the exact total/ETA stat bar
|
||||
- announce each operand on stderr before its passes (2026-07-24, branch
|
||||
`scan-operand-progress`): with per-operand walk/hash/update cycles, a
|
||||
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
|
||||
whole run's
|
||||
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
|
||||
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
|
||||
4h on a ZFS server); it is now a per-directory worker-pool traversal that
|
||||
records size/mtime during the walk (folding away the separate stat pass,
|
||||
halving metadata I/O), and each `PATH` operand commits in its own transaction
|
||||
so an interrupted scan keeps completed operands
|
||||
(2026-08-09, branch `cibuild-cache-bust`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/32)
|
||||
|
||||
- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
|
||||
now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
|
||||
disabled) keyed by absolute path that survives between runs — a rescan hashes
|
||||
only new or changed files (by mtime/size), deletes records for files vanished
|
||||
from under the scanned operands, and leaves records outside them untouched, so
|
||||
`scan` can be cronned daily; `report` and `trees` read the database (no
|
||||
positional arguments) instead of a scan stream. Database at
|
||||
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
|
||||
- `script/bootstrap` checked the installed `golangci-lint` version instead of
|
||||
only its presence. The version lived only in `GOLANGCI_LINT_VERSION`, with the
|
||||
`go install` module ref derived from it. Any installed version that was not
|
||||
the pin was reinstalled: older, newer, absent or unparseable. `go install`
|
||||
writes into `GOBIN` (or `GOPATH/bin`) while `make lint` ran the first
|
||||
`golangci-lint` on `PATH`, so bootstrap re-read the effective version after
|
||||
installing. On a mismatch it printed both paths and both versions and exited
|
||||
non-zero; it did not reorder `PATH` or delete anyone's binary. The `--version`
|
||||
call kept its stderr and was bounded by `timeout(1)` where that exists. `git`,
|
||||
`make` and `go` kept presence-only checks. `make check` and `make lint` were
|
||||
clean at v2.12.2, so v2.10.1 had not been hiding findings on `main`.
|
||||
Superseded by https://git.eeqj.de/sneak/sfdupes/issues/46: the linter is no
|
||||
longer installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
||||
|
||||
- The hash worker pool unwinds on the error path. The pool is an owned,
|
||||
context-aware `hashPool`. Every blocking send in the feeder and the workers
|
||||
selects on `ctx.Done()`, and `jobs` is closed on every path out. `hashPhase`
|
||||
defers `pool.stop()`, which cancels and then drains `results` until the last
|
||||
goroutine has exited; draining is what frees a worker parked on a send. `ctx`
|
||||
is threaded from `cmd.Context()` through `runScan`, `syncScan`, both worker
|
||||
pools and the whole database layer, as the first parameter everywhere. The
|
||||
walk pool gets the same treatment, plus a `ctx.Err()` guard after the walk. A
|
||||
cancelled walk yields a partial size census, and every file it never reached
|
||||
looks vanished to the update phase. That phase's own `BeginTx` also fails on
|
||||
the cancelled context before deleting anything, but the guard is the barrier
|
||||
that still holds once an interrupted scan may commit what it has. Tests drive
|
||||
`run(scan)` against a database whose insert trigger aborts, and assert that
|
||||
the scan fails instead of hanging and that the goroutine count returns to its
|
||||
pre-scan baseline. Other tests cancel a scan part-way through the walk and
|
||||
assert that it stops at the guard with every record intact. Direct tests of
|
||||
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
|
||||
and `hashPhase` cover the remaining cancellation branches of both pools
|
||||
(2026-08-09, branch `hash-pool-cleanup`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/6)
|
||||
|
||||
- The database is closed on every fatal exit path. `fatalf` and its `os.Exit(1)`
|
||||
are gone, so the deferred `db.Close()`, and with it the SQLite WAL checkpoint,
|
||||
runs when a subcommand fails. `runScan`, `runReport`, `runTrees`,
|
||||
`loadRecords` and `resolveRoots` return errors instead. The single exit point
|
||||
is `run` in `main.go`. It maps a `fatalError` to exit 1 and cobra's own
|
||||
argument and flag errors to exit 2, so a runtime failure is never reported as
|
||||
a usage error or printed with the usage text. `main_test.go` drives the CLI
|
||||
in-process and asserts the exit codes from README "Error handling" and the
|
||||
stdout/stderr split. That includes a fatal error after the database is open
|
||||
leaving no `-wal` or `-shm` file behind, for `scan`, `report` or `trees`
|
||||
(2026-08-09, branch `db-close-on-fatal`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||
|
||||
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as
|
||||
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
|
||||
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
|
||||
replaced with the canonical file. The linter settings (`lll`, `funlen`,
|
||||
`cyclop`, `dupl` thresholds) now live under `linters.settings`, per the v2
|
||||
schema, so they are applied. No new lint findings surfaced (2026-08-09, branch
|
||||
`golangci-v2.12.2`, https://git.eeqj.de/sneak/sfdupes/issues/3)
|
||||
|
||||
- Makefile targets converted to scripts-to-rule-them-all `script/` entrypoints,
|
||||
like the other managed repos. All 12 entrypoints exist:
|
||||
- `bootstrap`, `setup` and `projectname`;
|
||||
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
|
||||
- `docker`, `cibuild`, `precommit` and `install-precommit`.
|
||||
|
||||
Every Makefile target is a thin shim over them (2026-07-26, commit
|
||||
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
|
||||
|
||||
- The binary is the default Make target: plain `make` builds `sfdupes`, where it
|
||||
used to run `check` plus `build`. `make build` remains as an alias
|
||||
(2026-07-24, branch `make-default-target`)
|
||||
|
||||
- Scan-wide phases, concurrent operands and batched updates. All operands seed
|
||||
the shared walk pool, and every pass runs once over the whole scan, so totals
|
||||
and ETAs are scan-global. The per-operand walk, hash and update cycles and
|
||||
their stderr announcements are gone. The update pass commits in batched
|
||||
transactions. The filesystem is authoritative and the database an
|
||||
eventually-consistent reflection of it, so scan-level atomicity is not
|
||||
required (2026-07-24, branch `scan-wide-phases`)
|
||||
|
||||
- The stat pass split back out of the walk. Phases are strictly sequential
|
||||
again, per operand, with parallelism only inside each phase:
|
||||
- walk;
|
||||
- stat;
|
||||
- hash;
|
||||
- update.
|
||||
|
||||
The walk enumerates paths with per-directory workers, and the stat pass
|
||||
lstats them with per-file workers, restoring the exact total and ETA of the
|
||||
stat bar (2026-07-24, branch `parallel-phases`)
|
||||
|
||||
- Each operand is announced on stderr before its passes. With per-operand walk,
|
||||
hash and update cycles, a multi-operand run such as `scan /srv/*` showed pass
|
||||
totals that looked like the whole run's (2026-07-24, branch
|
||||
`scan-operand-progress`)
|
||||
|
||||
- Parallel walk. The walk pass was a single goroutine and took hours at about 20
|
||||
million files on a busy pool: 22 million files in 4 hours on a ZFS server. It
|
||||
is now a per-directory worker-pool traversal that records size and mtime
|
||||
during the walk, folding away the separate stat pass and halving metadata I/O.
|
||||
Each `PATH` operand commits in its own transaction, so an interrupted scan
|
||||
keeps completed operands (2026-07-24, branch `parallel-walk`)
|
||||
|
||||
- Persistent scan database. `scan` maintains a SQLite database
|
||||
(`modernc.org/sqlite`, pure Go, cgo stays disabled), keyed by absolute path,
|
||||
that survives between runs. A rescan hashes only new or changed files, by
|
||||
mtime and size. It deletes the records of files vanished from under the
|
||||
scanned operands and leaves records outside them untouched, so `scan` can be
|
||||
cronned daily. `report` and `trees` read the database, with no positional
|
||||
arguments, instead of a scan stream. The database is at
|
||||
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`. WAL
|
||||
journaling plus a single-transaction update keep a report run during a scan
|
||||
safe
|
||||
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
|
||||
and push `main` plus tags (2026-07-23)
|
||||
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
||||
`PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
|
||||
`-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
|
||||
boundaries, which are crossed by default
|
||||
- bring the repo into full policy compliance (2026-07-23, branch
|
||||
`repo-policy-compliance`; checklist below)
|
||||
- `git init` with README-only first commit; code baseline committed on `main`
|
||||
(2026-07-22)
|
||||
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
|
||||
safe (2026-07-24, branch `persistent-database`)
|
||||
|
||||
- The `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`) added, `v0.0.1`
|
||||
tagged, and `main` plus tags pushed (2026-07-23)
|
||||
|
||||
- `scan` CLI rework. Required `PATH...` operands via cobra replace the `/srv`
|
||||
`-root` default. The new `-x`/`--one-file-system` flag (GNU convention) stops
|
||||
at filesystem boundaries, which are crossed by default (2026-07-23, branch
|
||||
`scan-required-paths`)
|
||||
|
||||
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
|
||||
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
|
||||
2026-07-22 found these gaps, and the branch closed them:
|
||||
- `.gitignore`, with OS, editor, Go and secrets patterns; the compiled
|
||||
`sfdupes` binary and `files.dat` had been sitting untracked in the tree;
|
||||
- `.editorconfig`;
|
||||
- `LICENSE` and the README License section. MIT was assumed from house
|
||||
convention, for the owner to confirm;
|
||||
- `REPO_POLICIES.md` in the repo root;
|
||||
- the canonical `.golangci.yml`, with the 150 findings it raised fixed;
|
||||
- the `Makefile` targets `test`, `lint`, `fmt`, `fmt-check`, `docker` and
|
||||
`hooks`, and a `check` target that no longer depends on `build`;
|
||||
- a test suite, which the policy required with a 30-second timeout and the
|
||||
conditional `-v` rerun. It covers parsing, grouping, digests, suppression,
|
||||
hashing and the scan pipeline, at 64% coverage;
|
||||
- a multistage `Dockerfile` with hash-pinned images, a fail-fast lint stage
|
||||
and a build stage running `make check`, plus `.dockerignore`;
|
||||
- `.gitea/workflows/check.yml`, running `docker build .` on push with the
|
||||
checkout action pinned by commit SHA;
|
||||
- the README sections the policy requires: a Description first line with the
|
||||
name, purpose, category, license and author, then Getting Started,
|
||||
Rationale, TODO, License and Author;
|
||||
- the stale README non-goal "no git repository setup and no CI" removed;
|
||||
- the pre-commit hook installed.
|
||||
|
||||
The code was already `gofmt`-clean, so no standalone formatting commit was
|
||||
needed (2026-07-23, branch `repo-policy-compliance`)
|
||||
|
||||
- `git init` with a README-only first commit; the code baseline committed on
|
||||
`main` (2026-07-22)
|
||||
|
||||
- `scan`, `report` and `trees` subcommands implemented (pre-git history)
|
||||
|
||||
# Future Steps
|
||||
|
||||
- possible later features (explicitly out of scope per README): full-content
|
||||
verification of candidates, removal-script helpers
|
||||
- Possible later features, out of scope per README: full-content verification of
|
||||
candidates, and helper scripts for removal.
|
||||
|
||||
# Repo Policy Compliance
|
||||
# Accepted Divergences
|
||||
|
||||
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
|
||||
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
|
||||
formatting commit is needed.
|
||||
|
||||
- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
|
||||
untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
|
||||
- [x] `.editorconfig` missing
|
||||
- [x] `LICENSE` missing and README has no License section (MIT assumed from
|
||||
house convention — user to confirm)
|
||||
- [x] `REPO_POLICIES.md` missing from repo root
|
||||
- [x] `.golangci.yml` missing (install canonical copy); code must then pass
|
||||
`make lint` (150 findings fixed; `make lint` is clean)
|
||||
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
|
||||
`docker`, `hooks`; `check` currently depends on `build`, which writes the
|
||||
binary (`make check` must not modify files)
|
||||
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
|
||||
with a 30-second timeout and the conditional `-v` rerun pattern (suite
|
||||
covers parsing, grouping, digests, suppression, hashing, and the scan
|
||||
pipeline; 64% coverage)
|
||||
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
|
||||
lint stage, build stage running `make check`
|
||||
- [x] `.dockerignore` missing
|
||||
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
|
||||
action pinned by commit SHA)
|
||||
- [x] README lacks required sections: Description first line
|
||||
(name/purpose/category/license/author), Getting Started, Rationale, TODO,
|
||||
License, Author
|
||||
- [x] README non-goal "no git repository setup and no CI" is stale now that the
|
||||
repo is under git with CI
|
||||
- [x] pre-commit hook not installed (`make hooks` once the target exists)
|
||||
|
||||
Accepted divergences (no action):
|
||||
|
||||
- flat single-package layout with `.go` files in the repo root — fine for a
|
||||
small single-binary tool per the Go styleguide; the tracker audit agrees
|
||||
- Flat single-package layout, with the `.go` files in the repo root. That is
|
||||
fine for a small single-binary tool per the Go styleguide, and the tracker
|
||||
audit agrees.
|
||||
|
||||
Reference in New Issue
Block a user