Author SHA1 Message Date
clawbot 282146a409 Reformat all Markdown with prettier
check / check (push) Successful in 2m2s
Mechanical result of `make fmt` now that prettier runs over Markdown:
README.md and TODO.md rewrapped to the house settings (4-space,
proseWrap always). REPO_POLICIES.md was already compliant. No prose was
changed by hand in this commit.

Model: opus-4-8
2026-09-21 07:32:12 +00:00
clawbot 2dd4b8c401 Restore Markdown formatting in fmt/fmt-check and pin prettier (closes #19)
script/fmt and script/fmt-check now run gofmt for Go and prettier for
Markdown; fmt-check reports each independently. prettier is pinned at
3.8.1 by package.json/yarn.lock (integrity hash); .prettierrc and
.prettierignore are the house settings copied from the prompts repo.
script/bootstrap installs node and yarn from the host package manager
and runs `yarn install --frozen-lockfile`.

The Markdown check runs in CI via the Dockerfile build stage, where
bootstrap provides prettier; it is removed from the lint stage because
the golangci-lint image has no node. JS manifests are copied before
bootstrap so the yarn layer caches, and node_modules is dockerignored.

node is an unpinned host runtime like git/make/go: nvm's glibc node does
not run on the musl/Alpine build image, so the canonical nvm route is
not usable here; prettier is the hash-pinned formatter. The wholesale
Markdown reformat follows in the next commit.

Model: opus-4-8
2026-09-21 07:31:53 +00:00
13 changed files with 832 additions and 846 deletions
+2
View File
@@ -2,6 +2,8 @@
.claude
.DS_Store
sfdupes
files.dat
node_modules
*.log
*.out
*.test
+1
View File
@@ -27,6 +27,7 @@ node_modules/
*.log
# Local scan data
files.dat
*.sqlite
*.sqlite-shm
*.sqlite-wal
+2
View File
@@ -0,0 +1,2 @@
node_modules/
yarn.lock
+4
View File
@@ -0,0 +1,4 @@
{
"tabWidth": 4,
"proseWrap": "always"
}
+11 -6
View File
@@ -27,9 +27,12 @@ ARG CHECK_EPOCH
# target now runs `docker build -f Dockerfile.lint`, and a docker build
# cannot run a docker build: routing the gate through make would mean
# nesting docker inside this image. Same reason `make check` is gone
# from the build stage below. `make fmt-check` stays as it is — it is a
# gate, not the aggregate, and it shells out to nothing.
RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
# from the build stage below.
#
# `make fmt-check` is not run in this stage: it now also runs prettier
# over Markdown, and this golangci-lint image has no node. The gate runs
# in the build stage below, where script/bootstrap installs node and
# prettier.
# The FROM above and the one in Dockerfile.lint pin the same linter
# twice, and nothing else keeps them in sync; this fails the build when
@@ -77,10 +80,12 @@ COPY --from=lint /src/go.sum /dev/null
# rather than duplicating the installs inline. Only script/ and the
# dependency manifests are copied first, nothing else, so this layer
# stays cached until the scripts or the dependencies change — bootstrap
# ends in `go mod download`, which is why there is no separate
# invocation of it here.
# runs `go mod download` and `yarn install`, which is why there is no
# separate invocation of either here. The JS manifests (package.json,
# yarn.lock) are copied too so the yarn install layer caches alongside
# the Go one.
COPY script/ script/
COPY go.mod go.sum ./
COPY go.mod go.sum package.json yarn.lock ./
RUN script/bootstrap
COPY . .
+1 -1
View File
@@ -46,4 +46,4 @@ hooks:
@script/install-precommit
clean:
rm -f $(BINARY)
rm -f $(BINARY) files.dat
+384 -437
View File
@@ -2,19 +2,18 @@
## Description
`sfdupes` is an MIT-licensed Go CLI tool by
[@sneak](https://sneak.berlin) that quickly identifies *candidate*
duplicate files — and, ultimately, entire duplicate directory trees —
across very large filesystems without reading full file contents. Files
are considered duplicates when they have identical size, identical
SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
1024 bytes. This is a strong candidate signal, not proof of identical
content (the middle of the file is never read); the intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte is prohibitively
expensive. `scan` maintains a persistent SQLite database of file
signatures that survives between runs, so it can be run from cron and
the reports can be generated at any time from the most recent scan.
`sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that
quickly identifies _candidate_ duplicate files — and, ultimately, entire
duplicate directory trees — across very large filesystems without reading full
file contents. Files are considered duplicates when they have identical size,
identical SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
1024 bytes. This is a strong candidate signal, not proof of identical content
(the middle of the file is never read); the intended use is finding duplicate
downloads and duplicated directory trees on multi-terabyte ZFS servers where
reading every byte is prohibitively expensive. `scan` maintains a persistent
SQLite database of file signatures that survives between runs, so it can be run
from cron and the reports can be generated at any time from the most recent
scan.
This README is the complete and authoritative specification.
@@ -28,91 +27,81 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes trees > dupetrees.tsv
```
`scan` walks one or more filesystem trees and maintains one database
record per regular file (path, size, mtime, head hash, tail hash). The
database persists between runs; a rescan only hashes files that are new
or changed, and removes records for files that no longer exist.
`report` reads the database and prints the file-level duplicates
report. `trees` reads the same database and prints the duplicate-tree
report. A missing/invalid subcommand — or a `scan` invocation with no
`PATH` operand — prints a usage message and exits 2.
`scan` walks one or more filesystem trees and maintains one database record per
regular file (path, size, mtime, head hash, tail hash). The database persists
between runs; a rescan only hashes files that are new or changed, and removes
records for files that no longer exist. `report` reads the database and prints
the file-level duplicates report. `trees` reads the same database and prints the
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
with no `PATH` operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed
anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a
daily `sfdupes scan` cron job, with the reporting commands run
interactively whenever needed; their results are as fresh as the last
completed scan.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
cron job, with the reporting commands run interactively whenever needed; their
results are as fresh as the last completed scan.
## Rationale
Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
only from files whose size at least one other file shares, since a
size-unique file cannot be a duplicate — makes a full-filesystem sweep
tractable, and the signatures are kept in a persistent database, so
the expensive filesystem pass is incremental: a rescan re-hashes only
files whose recorded mtime or size changed, and all analysis happens
offline from the database alone. The end goal is
not individual files but whole duplicated trees — duplicate
extractions, duplicate downloads, copied project trees — which an
operator can consider removing as a unit.
Duplicate finders that hash entire files do not scale to the target environment:
~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under
resilver). Reading at most 2 KiB per file — and only from files whose size at
least one other file shares, since a size-unique file cannot be a duplicate —
makes a full-filesystem sweep tractable, and the signatures are kept in a
persistent database, so the expensive filesystem pass is incremental: a rescan
re-hashes only files whose recorded mtime or size changed, and all analysis
happens offline from the database alone. The end goal is not individual files
but whole duplicated trees — duplicate extractions, duplicate downloads, copied
project trees — which an operator can consider removing as a unit.
## Design
Goals, in order:
1. **Find whole duplicate trees, not just files.** The end goal is to
identify places where the exact same set of files and directories
exists at two or more paths (duplicate extractions, duplicate
downloads, copied project trees), so the operator can consider
removing an entire subtree at once. File-level duplicate detection is
the foundation; tree-level detection is built on top of it.
2. **Never read full file contents.** At most 2 KiB is read per file
(first and last 1024 bytes), and only files whose size at least
one other file shares are read at all — a size-unique file cannot
be a duplicate. Scale target: tens of millions of files, ~150 TB
filesystem, possibly slow or busy disks (ZFS pool under resilver).
Holding one small record (path, size, mtime) per file in memory
during a scan is acceptable; holding every file's hashes is not
(they stay in the database).
3. **Scan incrementally, analyze offline.** The expensive filesystem
scan maintains a persistent database; an unchanged file is never
read again on a rescan. All analysis (`report`, `trees`) works from
the database alone and must never touch the scanned filesystem
again. `scan` is designed to be cronned; the reports run at any
time against the last completed scan.
4. **Clean stream separation.** Everything on stdout is machine-readable
data. All progress, warnings, and summaries go to stderr. Never mix
them.
1. **Find whole duplicate trees, not just files.** The end goal is to identify
places where the exact same set of files and directories exists at two or
more paths (duplicate extractions, duplicate downloads, copied project
trees), so the operator can consider removing an entire subtree at once.
File-level duplicate detection is the foundation; tree-level detection is
built on top of it.
2. **Never read full file contents.** At most 2 KiB is read per file (first and
last 1024 bytes), and only files whose size at least one other file shares
are read at all — a size-unique file cannot be a duplicate. Scale target:
tens of millions of files, ~150 TB filesystem, possibly slow or busy disks
(ZFS pool under resilver). Holding one small record (path, size, mtime) per
file in memory during a scan is acceptable; holding every file's hashes is
not (they stay in the database).
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
maintains a persistent database; an unchanged file is never read again on a
rescan. All analysis (`report`, `trees`) works from the database alone and
must never touch the scanned filesystem again. `scan` is designed to be
cronned; the reports run at any time against the last completed scan.
4. **Clean stream separation.** Everything on stdout is machine-readable data.
All progress, warnings, and summaries go to stderr. Never mix them.
### Constraints
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
`sfdupes`.
- Dependencies: standard library, `github.com/spf13/cobra` for the
CLI, **one progress-bar library**
(`github.com/schollz/progressbar/v3`), and **one SQLite driver**
(`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
`github.com/spf13/viper` is permitted if configuration-file support
is ever needed, but is not currently used. No other third-party
deps.
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`.
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, **one
progress-bar library** (`github.com/schollz/progressbar/v3`), and **one SQLite
driver** (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
`github.com/spf13/viper` is permitted if configuration-file support is ever
needed, but is not currently used. No other third-party deps.
- Cross-compilation is not a concern. Builds run with cgo disabled (the
`Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
- Analysis modes (`report`, `trees`) must be deterministic: identical
database contents, identical output, regardless of the order in
which records were inserted.
- Analysis modes (`report`, `trees`) must be deterministic: identical database
contents, identical output, regardless of the order in which records were
inserted.
### Subcommands
Three subcommands, all implemented:
1. `scan` — walk the filesystem and synchronize the database: one
signature record per regular file.
1. `scan` — walk the filesystem and synchronize the database: one signature
record per regular file.
2. `report` — file-level duplicate report from the database.
3. `trees` — tree-level duplicate report: reconstruct the directory
hierarchy from the database records, compute a Merkle-style digest
per directory, and report maximal groups of identical trees.
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
from the database records, compute a Merkle-style digest per directory, and
report maximal groups of identical trees.
```
sfdupes scan [--workers N] [-x] PATH...
@@ -124,25 +113,22 @@ sfdupes trees > dupetrees.tsv
All three subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable
when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`.
There is no command-line flag.
- `scan` creates the database (and its parent directory) on first
use. `report` and `trees` require an existing database; a missing
database file is a fatal error (exit 1) telling the user to run
`scan` first.
- The database uses WAL journal mode and a busy timeout, so running a
report while a cron `scan` is in progress is safe. The filesystem
is authoritative; the database is an eventually-consistent
reflection of it. Hashed records are committed in batched
transactions while the scan is still running (keeping the WAL
small and letting concurrent reports observe progress), so a
report may see a scan's changes partially applied, and a scan
that dies partway leaves a valid database holding everything
hashed so far; the next scan skips those records and converges
toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
command-line flag.
- `scan` creates the database (and its parent directory) on first use. `report`
and `trees` require an existing database; a missing database file is a fatal
error (exit 1) telling the user to run `scan` first.
- The database uses WAL journal mode and a busy timeout, so running a report
while a cron `scan` is in progress is safe. The filesystem is authoritative;
the database is an eventually-consistent reflection of it. Hashed records are
committed in batched transactions while the scan is still running (keeping the
WAL small and letting concurrent reports observe progress), so a report may
see a scan's changes partially applied, and a scan that dies partway leaves a
valid database holding everything hashed so far; the next scan skips those
records and converges toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
with any other version is a fatal error):
```sql
CREATE TABLE files (
@@ -154,167 +140,147 @@ All three subcommands operate on a single SQLite database file:
) WITHOUT ROWID;
```
Paths are stored as BLOBs because Unix paths are raw bytes, not
guaranteed UTF-8. `mtime` is used only for change detection; it is
not part of the duplicate key. `head` and `tail` are empty strings
when the file has never been hashed because its size was unique as
of the last scan that covered it; such records still define the
file for tree reconstruction but never participate in duplicate
groups.
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
UTF-8. `mtime` is used only for change detection; it is not part of the
duplicate key. `head` and `tail` are empty strings when the file has never
been hashed because its size was unique as of the last scan that covered it;
such records still define the file for tree reconstruction but never
participate in duplicate groups.
### `scan` mode
`scan` requires one or more `PATH` operands naming the trees to scan.
There is no default path; invoking `scan` with no operand is a usage
error (usage message on stderr, exit 2). An operand may be a directory
or a regular file; an operand that does not exist is a fatal error
(exit 1). Because database records persist between runs and are keyed
by absolute path, each operand is resolved to an absolute, lexically
cleaned path (symlinks are not resolved) before walking, so results do
not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared
walk worker pool. Overlapping operands are harmless — an operand that
duplicates another or lies under another is dropped before walking,
so every file is reached exactly once and produces one database
record.
`scan` requires one or more `PATH` operands naming the trees to scan. There is
no default path; invoking `scan` with no operand is a usage error (usage message
on stderr, exit 2). An operand may be a directory or a regular file; an operand
that does not exist is a fatal error (exit 1). Because database records persist
between runs and are keyed by absolute path, each operand is resolved to an
absolute, lexically cleaned path (symlinks are not resolved) before walking, so
results do not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared walk worker
pool. Overlapping operands are harmless — an operand that duplicates another or
lies under another is dropped before walking, so every file is reached exactly
once and produces one database record.
`scan` synchronizes the database with the filesystem state under the
scanned operands:
`scan` synchronizes the database with the filesystem state under the scanned
operands:
- Only a file whose size at least one other file shares is ever
read: a size-unique file cannot be a duplicate, so it is recorded
without hashes (`head` and `tail` empty). The size census covers
every file walked this scan plus every database record outside
the scanned operands, so a possible duplicate of a separately
scanned tree is still recognized.
- A file not yet in the database is inserted: hashed when its size
is shared, without hashes otherwise.
- A file already in the database is **skipped without reading its
contents** when its lstat size equals the recorded size and its
lstat mtime is not newer than the recorded mtime. This is what
makes a daily rescan cheap. Exception: an unchanged file whose
record lacks hashes is hashed — and its record updated — once its
size becomes shared, so hashing deferred by size-uniqueness
happens as soon as it could matter.
- A file whose mtime is newer than recorded, or whose size differs,
is processed as if new: re-hashed, or recorded without hashes,
per the shared-size rule.
- A database record whose path lies under one of the scanned operands
but was not successfully processed this run is deleted. This
removes records for deleted files. It also removes records for
paths that failed to stat or hash this run: the database only ever
contains signatures verified by the most recent scan that covered
them (a subsequent successful scan re-adds such files).
- Database records outside the scanned operands are untouched, so
disjoint trees can be scanned on different schedules into the same
database.
- Only a file whose size at least one other file shares is ever read: a
size-unique file cannot be a duplicate, so it is recorded without hashes
(`head` and `tail` empty). The size census covers every file walked this scan
plus every database record outside the scanned operands, so a possible
duplicate of a separately scanned tree is still recognized.
- A file not yet in the database is inserted: hashed when its size is shared,
without hashes otherwise.
- A file already in the database is **skipped without reading its contents**
when its lstat size equals the recorded size and its lstat mtime is not newer
than the recorded mtime. This is what makes a daily rescan cheap. Exception:
an unchanged file whose record lacks hashes is hashed — and its record updated
— once its size becomes shared, so hashing deferred by size-uniqueness happens
as soon as it could matter.
- A file whose mtime is newer than recorded, or whose size differs, is processed
as if new: re-hashed, or recorded without hashes, per the shared-size rule.
- A database record whose path lies under one of the scanned operands but was
not successfully processed this run is deleted. This removes records for
deleted files. It also removes records for paths that failed to stat or hash
this run: the database only ever contains signatures verified by the most
recent scan that covered them (a subsequent successful scan re-adds such
files).
- Database records outside the scanned operands are untouched, so disjoint trees
can be scanned on different schedules into the same database.
`scan` runs **three sequential phases over the whole scan**.
Parallelism lives inside each phase; batched database writes begin
during the hash phase:
`scan` runs **three sequential phases over the whole scan**. Parallelism lives
inside each phase; batched database writes begin during the hash phase:
1. **walk + stat** — enumerate the trees under all `PATH` operands
concurrently with the walk worker pool: every operand seeds the
shared queue, and each worker reads one directory at a time,
handing discovered subdirectories back to the queue and running
`lstat` on each regular file as it is discovered (while the
directory's metadata is still hot). Sequential directory
enumeration is metadata-latency-bound and takes hours at tens of
millions of files; per-directory parallelism is what makes the
walk tractable on large or busy pools. The walk builds the size
census and resolves unchanged already-hashed files on the fly;
every other file is carried to the hash phase as a (path, size,
mtime) record.
2. **hash** — with the census complete, each carried file's size
decides its fate. Size-unique files are never read: new or
changed ones are recorded without hashes in the update phase,
unchanged unhashed ones simply keep their records. Every file
with a shared size is hashed by the worker pool: read the first
`min(1024, size)` bytes and the last `min(1024, size)` bytes
(one read when `size <= 1024`, since the two windows coincide)
and compute the SHA-256 of each. Zero-length files have constant
hashes and are never opened. Files are hashed in **inode order**
(minimizing seeks on spinning disks), and paths that are hard
links to the same inode are **read once**, all sharing the one
result — a hard-link backup farm costs one read per inode, not
per path. The phase total counts actual reads, so progress and
ETA are meaningful. Completed records are committed in batched
transactions **while hashing runs**, so a scan interrupted after
hours keeps everything hashed so far and the next scan resumes
cheaply, skipping records already written.
3. **update** — commit the final partial batch, the hash-less
records for size-unique new and changed files, and the deletions
for records the scan did not verify (vanished files, plus paths
that failed to stat or hash).
1. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
with the walk worker pool: every operand seeds the shared queue, and each
worker reads one directory at a time, handing discovered subdirectories back
to the queue and running `lstat` on each regular file as it is discovered
(while the directory's metadata is still hot). Sequential directory
enumeration is metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable on large
or busy pools. The walk builds the size census and resolves unchanged
already-hashed files on the fly; every other file is carried to the hash
phase as a (path, size, mtime) record.
2. **hash** — with the census complete, each carried file's size decides its
fate. Size-unique files are never read: new or changed ones are recorded
without hashes in the update phase, unchanged unhashed ones simply keep
their records. Every file with a shared size is hashed by the worker pool:
read the first `min(1024, size)` bytes and the last `min(1024, size)` bytes
(one read when `size <= 1024`, since the two windows coincide) and compute
the SHA-256 of each. Zero-length files have constant hashes and are never
opened. Files are hashed in **inode order** (minimizing seeks on spinning
disks), and paths that are hard links to the same inode are **read once**,
all sharing the one result — a hard-link backup farm costs one read per
inode, not per path. The phase total counts actual reads, so progress and
ETA are meaningful. Completed records are committed in batched transactions
**while hashing runs**, so a scan interrupted after hours keeps everything
hashed so far and the next scan resumes cheaply, skipping records already
written.
3. **update** — commit the final partial batch, the hash-less records for
size-unique new and changed files, and the deletions for records the scan
did not verify (vanished files, plus paths that failed to stat or hash).
Rules for the walk:
- Only regular files. Skip directories, symlinks (do not follow,
including symlink operands), sockets, FIFOs, and device nodes.
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
walking them would list every file once per snapshot).
- Filesystem boundaries are crossed by default. With `-x`
(long form `--one-file-system`, following the GNU `du`/`rsync`
convention), never descend into a directory on a different
filesystem than its `PATH` operand; each operand is bounded by its
own filesystem.
- On any per-path error (permission denied, file vanished between
passes, unreadable): print a one-line warning to stderr, skip the
path, and continue. Per-file errors never abort the run; the final
summary reports how many were skipped. As specified above, a
skipped path that has a database record from an earlier scan loses
that record; an unreadable directory subtree likewise loses its
records (accepted: the database mirrors what the latest scan could
- Only regular files. Skip directories, symlinks (do not follow, including
symlink operands), sockets, FIFOs, and device nodes.
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
them would list every file once per snapshot).
- Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each
operand is bounded by its own filesystem.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were
skipped. As specified above, a skipped path that has a database record from an
earlier scan loses that record; an unreadable directory subtree likewise loses
its records (accepted: the database mirrors what the latest scan could
actually verify).
Concurrency: the walk phase (which also stats files) and the hash
phase each use a worker pool of `--workers` workers (default
`runtime.NumCPU()`); the walk parallelizes across directories,
hashing across files. Both phases are seek-bound on spinning disks,
so raising `--workers` well past the core count can help on pools
with many spindles. The main goroutine owns partitioning, database
writes, and progress rendering; progress display must never block
the workers.
Concurrency: the walk phase (which also stats files) and the hash phase each use
a worker pool of `--workers` workers (default `runtime.NumCPU()`); the walk
parallelizes across directories, hashing across files. Both phases are
seek-bound on spinning disks, so raising `--workers` well past the core count
can help on pools with many spindles. The main goroutine owns partitioning,
database writes, and progress rendering; progress display must never block the
workers.
`scan` writes nothing to stdout. The summary line on stderr reports the
files seen this run broken down by disposition, plus skips:
`scan` writes nothing to stdout. The summary line on stderr reports the files
seen this run broken down by disposition, plus skips:
```
scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
```
(`removed` counts deleted database records, which are not part of the
files-seen total.)
(`removed` counts deleted database records, which are not part of the files-seen
total.)
### `report` mode
`report` reads every record from the database and takes no positional
arguments.
`report` reads every record from the database and takes no positional arguments.
**`report` must never touch the filesystem being analyzed.** It does not
stat, open, or otherwise access any path that appears in the records; its
only I/O is reading the database and writing stdout/stderr. It must
produce identical output whether or not the scanned filesystem is still
mounted.
**`report` must never touch the filesystem being analyzed.** It does not stat,
open, or otherwise access any path that appears in the records; its only I/O is
reading the database and writing stdout/stderr. It must produce identical output
whether or not the scanned filesystem is still mounted.
Processing:
- Records without hashes (size-unique when last scanned) are
excluded: their content is unknown, so they are never reported as
duplicates.
- Group the remaining records by the key
`(size, head_hash, tail_hash)`.
- Records without hashes (size-unique when last scanned) are excluded: their
content is unknown, so they are never reported as duplicates.
- Group the remaining records by the key `(size, head_hash, tail_hash)`.
- Every group with two or more paths is a duplicate group.
- Within each group, sort paths lexicographically (byte order). The
first path is the group's `first`; every other path is a `dupe`.
- Order groups by size descending (biggest reclaimable space first),
tie-broken by `first` path ascending. Output must be fully
deterministic for a given database state.
- Within each group, sort paths lexicographically (byte order). The first path
is the group's `first`; every other path is a `dupe`.
- Order groups by size descending (biggest reclaimable space first), tie-broken
by `first` path ascending. Output must be fully deterministic for a given
database state.
#### Report output format
TSV on stdout: a header line, then one row per duplicate file (N-1 rows
for a group of N):
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
group of N):
```
first dupe size
@@ -322,94 +288,86 @@ first dupe size
/srv/a/big.iso /srv/c/big-copy2.iso 4294967296
```
Summary to stderr: records read, number of duplicate groups, number of
dupe files, and total reclaimable bytes (sum of `size` over all dupe
rows) in human units.
Summary to stderr: records read, number of duplicate groups, number of dupe
files, and total reclaimable bytes (sum of `size` over all dupe rows) in human
units.
### `trees` mode
`trees` reads the same database as `report` (no positional arguments)
and reports **entire duplicate directory trees**: directories under
which the exact same set of relative paths exists with the exact same
file signatures.
`trees` reads the same database as `report` (no positional arguments) and
reports **entire duplicate directory trees**: directories under which the exact
same set of relative paths exists with the exact same file signatures.
**`trees` must never touch the filesystem being analyzed** — the same
rule as `report`. The directory hierarchy is reconstructed purely from
the paths in the records, split on `/`.
**`trees` must never touch the filesystem being analyzed** — the same rule as
`report`. The directory hierarchy is reconstructed purely from the paths in the
records, split on `/`.
Definitions:
- A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
informational and excluded. An unhashed record (empty hashes) has
unknown content: its signature is treated as unique to that file,
so a tree containing an unhashed file never compares equal to any
other tree.
- A directory's **digest** is a SHA-256 Merkle digest computed
bottom-up: serialize the directory's child entries — for a file
child, its name and signature; for a subdirectory child, its name
and that subdirectory's digest — sort the serialized entries
byte-lexicographically, and hash the concatenation. Names are part
of the digest: two trees whose files differ only in name are *not*
duplicates.
- Two directories are **duplicate trees** when their digests are
equal. Equal digests imply equal recursive file count and equal
total byte size.
informational and excluded. An unhashed record (empty hashes) has unknown
content: its signature is treated as unique to that file, so a tree containing
an unhashed file never compares equal to any other tree.
- A directory's **digest** is a SHA-256 Merkle digest computed bottom-up:
serialize the directory's child entries — for a file child, its name and
signature; for a subdirectory child, its name and that subdirectory's digest
sort the serialized entries byte-lexicographically, and hash the
concatenation. Names are part of the digest: two trees whose files differ only
in name are _not_ duplicates.
- Two directories are **duplicate trees** when their digests are equal. Equal
digests imply equal recursive file count and equal total byte size.
Known limitation (accepted): hard-linked paths are reported as
duplicates by `report` and count toward duplicate trees — their
content is genuinely identical — even though they share storage, so
removing one reclaims no space. Inode identity is used during the
scan to avoid redundant reads but is not persisted in the database.
Known limitation (accepted): hard-linked paths are reported as duplicates by
`report` and count toward duplicate trees — their content is genuinely identical
— even though they share storage, so removing one reclaims no space. Inode
identity is used during the scan to avoid redundant reads but is not persisted
in the database.
Known limitation (accepted): only regular files that appear in the
database define a tree. Empty directories are invisible, and a file
skipped during the scan (e.g. permission error) in one copy but not the
other will make otherwise-identical trees compare as different.
Known limitation (accepted): only regular files that appear in the database
define a tree. Empty directories are invisible, and a file skipped during the
scan (e.g. permission error) in one copy but not the other will make
otherwise-identical trees compare as different.
Processing:
- Build the hierarchy, compute every directory's digest, and group
directories by digest. Every group with two or more directories is a
duplicate-tree group.
- **Report only maximal trees.** A group is suppressed when its
members' parents are pairwise distinct directories that all share a
single digest — such a group is wholly implied by its parents' (or a
further ancestor's) group. Groups containing sibling directories, or
members whose parents differ, are always reported.
- Within each group, sort paths lexicographically (byte order); the
first path is `first`, every other path is a `dupe`.
- Order groups by total tree size descending, tie-broken by `first`
path ascending. Output must be fully deterministic for a given
input.
- Build the hierarchy, compute every directory's digest, and group directories
by digest. Every group with two or more directories is a duplicate-tree group.
- **Report only maximal trees.** A group is suppressed when its members' parents
are pairwise distinct directories that all share a single digest — such a
group is wholly implied by its parents' (or a further ancestor's) group.
Groups containing sibling directories, or members whose parents differ, are
always reported.
- Within each group, sort paths lexicographically (byte order); the first path
is `first`, every other path is a `dupe`.
- Order groups by total tree size descending, tie-broken by `first` path
ascending. Output must be fully deterministic for a given input.
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows
for a group of N). `files` is the recursive regular-file count of one
copy of the tree; `size` is the recursive total byte size of one copy:
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
group of N). `files` is the recursive regular-file count of one copy of the
tree; `size` is the recursive total byte size of one copy:
```
first dupe files size
/srv/a/project /srv/backup/project 3417 104857600
```
Summary to stderr: records read, number of duplicate-tree groups,
number of dupe trees, and total reclaimable bytes (sum of `size` over
all dupe rows) in human units.
Summary to stderr: records read, number of duplicate-tree groups, number of dupe
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
units.
### Progress
Use the progress-bar library for all scan progress; rendering in the
style of `pv` is the model. All progress goes to stderr.
Use the progress-bar library for all scan progress; rendering in the style of
`pv` is the model. All progress goes to stderr.
Each phase gets its own display, rendered the moment the phase
starts — a scan must never look hung. Loading the existing-record
index (`load`) and the walk have no known totals while running: show
a live count, rate, and elapsed time (spinner-style, no percentage or
ETA). The hash and update phases
have exact totals — only files that actually need hashing appear in
the hash total, so its ETA is meaningful. Required elements for the
bars with known totals:
Each phase gets its own display, rendered the moment the phase starts — a scan
must never look hung. Loading the existing-record index (`load`) and the walk
have no known totals while running: show a live count, rate, and elapsed time
(spinner-style, no percentage or ETA). The hash and update phases have exact
totals — only files that actually need hashing appear in the hash total, so its
ETA is meaningful. Required elements for the bars with known totals:
- elapsed time
- estimated time remaining
@@ -424,145 +382,136 @@ hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
Additional requirements:
- When stderr is not a TTY, do not emit ANSI redraws: print a plain
one-line progress update no more often than every 5 seconds instead.
- Progress updates are driven from the main goroutine and must be
non-blocking with respect to the worker pool.
- `report` and `trees` modes need no progress display, only their
stderr summaries.
- When stderr is not a TTY, do not emit ANSI redraws: print a plain one-line
progress update no more often than every 5 seconds instead.
- Progress updates are driven from the main goroutine and must be non-blocking
with respect to the worker pool.
- `report` and `trees` modes need no progress display, only their stderr
summaries.
### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., a `PATH` operand does not exist, the
database cannot be created/opened/read/written, a missing database
for `report`/`trees`, stdout write failure).
- `2`: usage error (including `scan` with no `PATH` operand and
`report`/`trees` with any positional argument).
- `1`: fatal error (e.g., a `PATH` operand does not exist, the database cannot
be created/opened/read/written, a missing database for `report`/`trees`,
stdout write failure).
- `2`: usage error (including `scan` with no `PATH` operand and `report`/`trees`
with any positional argument).
## Entrypoints
This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: the normalized executables in `script/` are the entrypoints
for the development workflow, and the `Makefile` targets are thin
shims that call them. Every script is POSIX `sh`, resolves the
repository root itself so it can be run from any working directory,
and may be invoked directly. The provided entrypoints are:
standard: the normalized executables in `script/` are the entrypoints for the
development workflow, and the `Makefile` targets are thin shims that call them.
Every script is POSIX `sh`, resolves the repository root itself so it can be run
from any working directory, and may be invoked directly. The provided
entrypoints are:
- `script/bootstrap` — install everything needed to build and
develop this repository, idempotently, assuming nothing is
present. `git`, `make`, and `go` come from the first of nix, apt,
brew, or apk found on the host, and are presence-checked only.
`golangci-lint` is deliberately **not** installed: it runs from a
digest-pinned image via `script/lint` and never from a host
install, so there is no host copy to drift from the pin. A missing
`docker` is warned about rather than installed or treated as
fatal — everything except linting works without it. Ends with
`go mod download`.
- `script/bootstrap` — install everything needed to build and develop this
repository, idempotently, assuming nothing is present. `git`, `make`, `go`,
and `node` come from the first of nix, apt, brew, or apk found on the host,
and are presence-checked only; `node` is an unpinned host runtime like the
rest, because nvm's prebuilt node is glibc-linked and does not run on this
repo's musl/Alpine build image. The Markdown formatter itself — `prettier` —
is pinned by `yarn.lock`'s integrity hash and installed with
`yarn install --frozen-lockfile`. `golangci-lint` is deliberately **not**
installed: it runs from a digest-pinned image via `script/lint` and never from
a host install, so there is no host copy to drift from the pin. A missing
`docker` is warned about rather than installed or treated as fatal —
everything except linting works without it. Ends with `go mod download` and
the `yarn` install.
- `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`.
- `script/projectname` — print this project's name (`sfdupes`).
Scripts that need the name call it, so they stay identical across
repositories.
- `script/test` — run the test suite with a 30-second timeout and
coverage enabled, rerunning verbosely on failure so the logs show
which test failed.
- `script/lint` — run the linter. It builds `Dockerfile.lint`, which
copies the repository into the digest-pinned
`golangci/golangci-lint` image and runs
`golangci-lint config verify` and `golangci-lint run` as build
steps, so a successful build is a clean lint. The linter is never
run on the host, which makes a working `docker` the one
prerequisite for linting — and therefore for `make check` and the
pre-commit hook. Offline machines: the gate steps themselves make
no network calls. `golangci-lint run` does not, and neither does
`golangci-lint config verify` — it validates against a schema the
pinned binary embeds, measured under `--network none` to both
pass a valid config and reject an invalid one. The build around
them does. `Dockerfile.lint` runs `go mod download` before the
gates and this module has external dependencies, so a first lint
on a machine with a cold BuildKit cache reaches the network there
(as well as pulling the pinned image); under `--network none` it
fails at that step, before any gate. That layer sits above the
gates and stays cached, so once it is warm `script/lint` — and
with it `make check` — runs entirely offline, until `go.mod` or
`go.sum` changes and the download layer goes cold again. Because
the daemon only ever sees a build context, this works when the
docker daemon is remote and bind mounts are impossible.
- `script/fmt` — format the Go sources in place (`gofmt -s -w`).
Markdown is not formatted.
- `script/fmt-check` — the read-only counterpart of `script/fmt`:
prints any unformatted file and exits non-zero instead of writing.
- `script/check` — run `script/test`, `script/lint`, and
`script/fmt-check`, in that order. Modifies nothing. Needs
`docker`, because `script/lint` does.
- `script/docker` — build the Docker image, tagged with the name
from `script/projectname`. The `Dockerfile` runs the gates as
build steps, so this is also the check a developer or reviewer
runs by hand.
- `script/cibuild` — build the Docker image untagged. This is what
the Gitea workflow runs on push; because the gates run as build
steps, a successful build implies the repository is green.
- `script/precommit` — run by the git pre-commit hook: `go mod tidy`
must be a no-op (a resulting change to `go.mod` or `go.sum` fails
the commit), then `script/check`.
- `script/install-precommit` — install the git pre-commit hook that
runs `script/precommit`. The hook is written to the common git
directory, so the main checkout and every worktree share it.
- `script/verify-lint-image-pin` — fail unless the
`golangci/golangci-lint` reference in `Dockerfile.lint` and the
one in the `Dockerfile` lint stage are the same image at the same
digest, naming both if not. The linter is pinned in those two
files and nothing else keeps them in sync, so a bump applied to
one alone would leave `make lint` and the `Dockerfile`'s
fail-fast lint stage checking the same tree against different
rulesets, both green. The guard restates neither pin — a third
copy would be the same drift one file further out — and runs as a
gate in both files, so `make lint`, `make check` and `make docker`
all catch it.
- `script/projectname` — print this project's name (`sfdupes`). Scripts that
need the name call it, so they stay identical across repositories.
- `script/test` — run the test suite with a 30-second timeout and coverage
enabled, rerunning verbosely on failure so the logs show which test failed.
- `script/lint` — run the linter. It builds `Dockerfile.lint`, which copies the
repository into the digest-pinned `golangci/golangci-lint` image and runs
`golangci-lint config verify` and `golangci-lint run` as build steps, so a
successful build is a clean lint. The linter is never run on the host, which
makes a working `docker` the one prerequisite for linting — and therefore for
`make check` and the pre-commit hook. Offline machines: the gate steps
themselves make no network calls. `golangci-lint run` does not, and neither
does `golangci-lint config verify` — it validates against a schema the pinned
binary embeds, measured under `--network none` to both pass a valid config and
reject an invalid one. The build around them does. `Dockerfile.lint` runs
`go mod download` before the gates and this module has external dependencies,
so a first lint on a machine with a cold BuildKit cache reaches the network
there (as well as pulling the pinned image); under `--network none` it fails
at that step, before any gate. That layer sits above the gates and stays
cached, so once it is warm `script/lint` — and with it `make check` — runs
entirely offline, until `go.mod` or `go.sum` changes and the download layer
goes cold again. Because the daemon only ever sees a build context, this works
when the docker daemon is remote and bind mounts are impossible.
- `script/fmt` — format in place: `gofmt -s -w` for Go sources and `prettier`
for Markdown (`--tab-width 4 --prose-wrap always`, the house settings, also
carried in `.prettierrc`). prettier is the pinned devDependency in
`package.json`/`yarn.lock`, installed by `script/bootstrap`.
- `script/fmt-check` — the read-only counterpart of `script/fmt`: runs both
checks, reports each independently so it is clear which failed, and exits
non-zero if either found unformatted files instead of writing.
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in
that order. Modifies nothing. Needs `docker`, because `script/lint` does.
- `script/docker` — build the Docker image, tagged with the name from
`script/projectname`. The `Dockerfile` runs the gates as build steps, so this
is also the check a developer or reviewer runs by hand.
- `script/cibuild` — build the Docker image untagged. This is what the Gitea
workflow runs on push; because the gates run as build steps, a successful
build implies the repository is green.
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a
no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then
`script/check`.
- `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit`. The hook is written to the common git directory, so the
main checkout and every worktree share it.
- `script/verify-lint-image-pin` — fail unless the `golangci/golangci-lint`
reference in `Dockerfile.lint` and the one in the `Dockerfile` lint stage are
the same image at the same digest, naming both if not. The linter is pinned in
those two files and nothing else keeps them in sync, so a bump applied to one
alone would leave `make lint` and the `Dockerfile`'s fail-fast lint stage
checking the same tree against different rulesets, both green. The guard
restates neither pin — a third copy would be the same drift one file further
out — and runs as a gate in both files, so `make lint`, `make check` and
`make docker` all catch it.
`script/verify-linter-pin` used to live here. It compared a linter
binary against a version pin in `script/bootstrap`, and both of its
subjects are gone: no linter binary is copied between build stages any
more, and bootstrap pins no version because it installs no linter. The
drift it existed to catch has moved from binary-versus-pin to
pin-versus-pin, which is what `script/verify-lint-image-pin` above
checks.
`script/verify-linter-pin` used to live here. It compared a linter binary
against a version pin in `script/bootstrap`, and both of its subjects are gone:
no linter binary is copied between build stages any more, and bootstrap pins no
version because it installs no linter. The drift it existed to catch has moved
from binary-versus-pin to pin-versus-pin, which is what
`script/verify-lint-image-pin` above checks.
`script/lint`, `script/docker` and `script/cibuild` all pass a freshly
computed `CHECK_EPOCH` build argument, and the gate steps in
`Dockerfile.lint` and `Dockerfile` reference it. Without that, an
unchanged tree lets Docker serve the gate layers from cache and the
build exits 0 having executed no tests and no lint — a green it never
earned, and one this repository has produced twice. `CHECK_EPOCH`
invalidates the gate layers on every run while leaving the pinned base
images and the dependency layers cached. `script/lint`'s value carries
the process id as well as the epoch, because two lint runs land inside
the same second easily and a bare epoch would cache the second one.
`script/lint`, `script/docker` and `script/cibuild` all pass a freshly computed
`CHECK_EPOCH` build argument, and the gate steps in `Dockerfile.lint` and
`Dockerfile` reference it. Without that, an unchanged tree lets Docker serve the
gate layers from cache and the build exits 0 having executed no tests and no
lint — a green it never earned, and one this repository has produced twice.
`CHECK_EPOCH` invalidates the gate layers on every run while leaving the pinned
base images and the dependency layers cached. `script/lint`'s value carries the
process id as well as the epoch, because two lint runs land inside the same
second easily and a bare epoch would cache the second one.
## Build
The `script/` entrypoints above are where the implementations live;
the `Makefile` targets are shims onto them, except `build`, which
carries the compile recipe:
The `script/` entrypoints above are where the implementations live; the
`Makefile` targets are shims onto them, except `build`, which carries the
compile recipe:
- `make` / `make build` — build the `sfdupes` binary (cgo
disabled); building is the default target.
- `make bootstrap` — install the build and development
dependencies.
- `make setup` — prepare a fresh clone: `bootstrap` plus the
pre-commit hook.
- `make test` — run the test suite (30-second timeout; reruns with
`-v` on failure).
- `make lint` — run `golangci-lint` with the repo config, in Docker
(see `script/lint`); requires `docker`.
- `make fmt` / `make fmt-check` — format Go sources / verify
formatting without writing.
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing.
Requires `docker`, via `lint`.
- `make docker` — build the Docker image, which runs the gates as
build stages.
- `make` / `make build` — build the `sfdupes` binary (cgo disabled); building is
the default target.
- `make bootstrap` — install the build and development dependencies.
- `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook.
- `make test` — run the test suite (30-second timeout; reruns with `-v` on
failure).
- `make lint` — run `golangci-lint` with the repo config, in Docker (see
`script/lint`); requires `docker`.
- `make fmt` / `make fmt-check` — format Go and Markdown sources / verify both
without writing.
- `make check` `test`, `lint`, and `fmt-check`; modifies nothing. Requires
`docker`, via `lint`.
- `make docker` — build the Docker image, which runs the gates as build stages.
- `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary.
@@ -572,8 +521,8 @@ All of the following, run in this directory, must pass:
1. `make check` passes (tests, lint, `gofmt`).
2. `make docker` succeeds.
3. Smoke test — create a throwaway tree in a temp dir (never test
against real data):
3. Smoke test — create a throwaway tree in a temp dir (never test against real
data):
```sh
d=$(mktemp -d)
@@ -605,30 +554,29 @@ All of the following, run in this directory, must pass:
./sfdupes report
```
(The scan database lives inside `$d` here purely for test hygiene;
scanning `$d` therefore also records the SQLite file itself, which
is harmless.)
(The scan database lives inside `$d` here purely for test hygiene; scanning
`$d` therefore also records the SQLite file itself, which is harmless.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin`
form one group (two dupe rows, `first` is the lexicographically
smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group;
`t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group;
`tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and
`tiny3` appear nowhere; groups ordered by size descending.
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
group (two dupe rows, `first` is the lexicographically smallest path);
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
`t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
`empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; groups
ordered by size descending.
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
`$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is
suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
appears nowhere (its file set differs by name).
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
differs by name).
Expected from the second `report` (after the modify/delete rescan):
`one.bin` has left its group (its content changed), so
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is
gone from the database.
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is gone from the
database.
The test suite automates this scenario (see `scan_test.go`), plus a
negative check: `report` and `trees` operate on the database alone
and never touch the scanned filesystem.
The test suite automates this scenario (see `scan_test.go`), plus a negative
check: `report` and `trees` operate on the database alone and never touch
the scanned filesystem.
## TODO
@@ -636,11 +584,10 @@ Tracked in [TODO.md](TODO.md).
## Non-goals
- No full-content verification, no byte-for-byte compare, no deletion
or linking of duplicates. The reports are advisory; acting on them is
the user's job.
- No persistence beyond the SQLite database described above; no
export/import formats.
- No full-content verification, no byte-for-byte compare, no deletion or linking
of duplicates. The reports are advisory; acting on them is the user's job.
- No persistence beyond the SQLite database described above; no export/import
formats.
- No daemon or filesystem watcher; scheduling rescans is cron's job.
## License
+335 -384
View File
@@ -1,435 +1,386 @@
# Workflow
- take an issue from the `1.0.0` milestone on the tracker; work not
yet on the tracker gets filed as an issue first
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
tracker gets filed as an issue first
- branch (from `main`)
- do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the
same commit as the work)
- push the branch and open a PR whose title ends with
` (closes #N)`
- an independent review gates the merge; every finding is addressed
or explicitly rebutted on the PR
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
as the work)
- push the branch and open a PR whose title ends with ` (closes #N)`
- an independent review gates the merge; every finding is addressed or
explicitly rebutted on the PR
- merge to `main` once the review passes
# Status
- pre-1.0
- the Gitea tracker is authoritative for the pre-1.0 backlog: the
open issues under the `1.0.0` milestone are what remains before
the tag, and this file records history and process, not the queue
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
under the `1.0.0` milestone are what remains before the tag, and this file
records history and process, not the queue
# Next Step
- take the next issue from the `1.0.0` milestone on the tracker:
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
the source of truth for what is left before 1.0.0. Individual
issues are deliberately not restated here; a copy in this file
drifts out of date the moment the tracker moves
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
of truth for what is left before 1.0.0. Individual issues are deliberately not
restated here; a copy in this file drifts out of date the moment the tracker
moves
# Completed Steps
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
and `.dockerignore` (2026-09-21, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/22)
- restore Markdown formatting in `script/fmt`/`fmt-check` and reformat all
Markdown to the house prettier settings (2026-09-21, closes
https://git.eeqj.de/sneak/sfdupes/issues/19)
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` (2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the
redundant tag, so both pins are the policy `# image:vX.Y.Z,
YYYY-MM-DD` comment over a bare `FROM image@sha256:...`. Digest
unchanged. `script/verify-lint-image-pin` parses those `FROM` lines
and still matches the tagless form; its advice line lost the now
meaningless "tag and digest". With no tag in either reference, a
tag-only disagreement no longer exists — a one-sided tag is caught as
a plain mismatch.
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag,
so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare
`FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin`
parses those `FROM` lines and still matches the tagless form; its advice line
lost the now meaningless "tag and digest". With no tag in either reference, a
tag-only disagreement no longer exists — a one-sided tag is caught as a plain
mismatch.
- run all linting in Docker via `Dockerfile.lint` and `script/lint`
(2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling, the
linter runs inside a container invoked through the `script/`
entrypoint and is never installed on a host. New root
`Dockerfile.lint` COPYs the repo into the digest-pinned
`golangci/golangci-lint:v2.12.2` image and runs
`golangci-lint config verify` and `golangci-lint run` as build
steps, so a successful build IS a clean lint; `script/lint` is
reduced to building it. `script/bootstrap` loses the `go install`,
the pin constants, the version parser and `verify_golangci_lint`
outright rather than hardening them — with nothing linting on the
host, the `$GOPATH/bin` versus `PATH` problem that motivated them has
no subject — and now warns rather than fails when `docker` is absent.
Two traps handled. A lint build on an unchanged tree returns success
in well under a second having run no linter, which is
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
owner ruling, the linter runs inside a container invoked through the `script/`
entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
runs `golangci-lint config verify` and `golangci-lint run` as build steps, so
a successful build IS a clean lint; `script/lint` is reduced to building it.
`script/bootstrap` loses the `go install`, the pin constants, the version
parser and `verify_golangci_lint` outright rather than hardening them — with
nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that
motivated them has no subject — and now warns rather than fails when `docker`
is absent. Two traps handled. A lint build on an unchanged tree returns
success in well under a second having run no linter, which is
https://git.eeqj.de/sneak/sfdupes/issues/32 and
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so
`Dockerfile.lint` carries `ARG CHECK_EPOCH` referenced
inside every gate `RUN` (BuildKit hashes the expanded command, not
the declaration) and `script/lint` passes `"$(date +%s)-$$"` — the
PID matters because two lint runs land inside the same second easily.
And nothing inside an image build may shell out to docker, so the
main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint`
carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes
the expanded command, not the declaration) and `script/lint` passes
`"$(date +%s)-$$"` the PID matters because two lint runs land inside the
same second easily. And nothing inside an image build may shell out to docker,
so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
instead of `make lint`, and its build stage runs `make test` and
`make fmt-check` instead of the `make check` aggregate (`make`, not
the scripts bare, because the Makefile's `export CGO_ENABLED = 0`
only reaches what it invokes). `COPY --from=lint`
`/usr/bin/golangci-lint` is replaced by
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the
only edge forcing BuildKit to finish linting before the build stage
starts, and dropping it without replacing the edge would have ended
fail-fast linting silently under a still-green build. That is
canonical `REPO_POLICIES.md:107`'s ordering edge, restored.
`ENV PATH=/home/builder/go/bin:$PATH` is gone with the `go install`
that justified it. `script/verify-linter-pin` is retired, deleted
along with its README entry, because both of its subjects ceased to
exist in the same change: it compared a linter binary against
`GOLANGCI_LINT_VERSION` in `script/bootstrap`, and there is now
neither a binary crossing between stages nor a version pin in
bootstrap. The drift it guarded has not gone away, it has moved — the
linter is still pinned twice, now as the `FROM` line of
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage,
with nothing syncing them, which is exactly what
`make fmt-check` instead of the `make check` aggregate (`make`, not the
scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches
what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge
forcing BuildKit to finish linting before the build stage starts, and dropping
it without replacing the edge would have ended fail-fast linting silently
under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering
edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the
`go install` that justified it. `script/verify-linter-pin` is retired, deleted
along with its README entry, because both of its subjects ceased to exist in
the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION`
in `script/bootstrap`, and there is now neither a binary crossing between
stages nor a version pin in bootstrap. The drift it guarded has not gone away,
it has moved — the linter is still pinned twice, now as the `FROM` line of
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with
nothing syncing them, which is exactly what
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
replacement is one new `script/verify-lint-image-pin`,
run as a gate in both files, which compares the two references to
each other and deliberately restates neither: a hardcoded expected
digest would be a third copy and the same drift one file further out.
`golangci-lint config verify` is included per the ruling, and the
concern about its unpinned live HTTPS schema fetch was measured
rather than assumed — under `--network none` the pinned binary both
passes a valid config and rejects an invalid one with the jsonschema
error, so it validates from an embedded schema and makes no network
call of its own. The README scopes that to the gate steps rather
than to linting as a whole: `Dockerfile.lint` runs `go mod download`
above them, so a cold cache still needs the network and only a warm
one lints offline. Verified: `make lint` green with every `PATH`
directory containing a `golangci-lint` removed
replacement is one new `script/verify-lint-image-pin`, run as a gate in both
files, which compares the two references to each other and deliberately
restates neither: a hardcoded expected digest would be a third copy and the
same drift one file further out. `golangci-lint config verify` is included per
the ruling, and the concern about its unpinned live HTTPS schema fetch was
measured rather than assumed — under `--network none` the pinned binary both
passes a valid config and rejects an invalid one with the jsonschema error, so
it validates from an embedded schema and makes no network call of its own. The
README scopes that to the gate steps rather than to linting as a whole:
`Dockerfile.lint` runs `go mod download` above them, so a cold cache still
needs the network and only a warm one lints offline. Verified: `make lint`
green with every `PATH` directory containing a `golangci-lint` removed
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
`command -v golangci-lint` empty); two consecutive `script/lint` runs
on an untouched tree both executed the linter, 27.7s and 28.7s in the
lint step under distinct epochs with the `COPY . .` layer `CACHED`
above them, at 42.2s and 41.8s wall clock — the no-cache rule was not
weakened to shorten that. Negative control: a planted
`var unusedIssue46Sentinel = 1` failed `script/lint` with
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and
failed `make docker` at `[lint 9/9]` with the build stage stopped at
`[builder 3/12]``COPY --from=lint`, `script/bootstrap`, the test
gate and `make build` all zero occurrences — then reverted clean. The
drift guard fails on a tag-only disagreement, on a digest-only
disagreement, and on an unreadable reference, naming both sides.
`make docker` green in 5m35s with all six gates executing under one
epoch (lint 37.6s, test 25.2s reporting
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`).
The non-root quirk still holds: in the builder image with the Go test
cache off, `--user 0:0` fails `TestScanHardlinkRunFailsTogether`
(exit 1) where the unprivileged user passes (exit 0). Noted for
follow-up, not fixed here: `golangci-lint` warns that the
`gomodguard` linter is deprecated since v2.12.0 in favour of
`gomodguard_v2`.
`command -v golangci-lint` empty); two consecutive `script/lint` runs on an
untouched tree both executed the linter, 27.7s and 28.7s in the lint step
under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s
and 41.8s wall clock — the no-cache rule was not weakened to shorten that.
Negative control: a planted `var unusedIssue46Sentinel = 1` failed
`script/lint` with
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed
`make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]`
`COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all
zero occurrences — then reverted clean. The drift guard fails on a tag-only
disagreement, on a digest-only disagreement, and on an unreadable reference,
naming both sides. `make docker` green in 5m35s with all six gates executing
under one epoch (lint 37.6s, test 25.2s reporting
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The
non-root quirk still holds: in the builder image with the Go test cache off,
`--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the
unprivileged user passes (exit 0). Noted for follow-up, not fixed here:
`golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0
in favour of `gomodguard_v2`.
- install the Docker build stage's prerequisites by running
`script/bootstrap` instead of `apk add --no-cache make` inline
(2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical
`REPO_POLICIES.md:97` requires it, and the inline install left the
build stage maintaining its own notion of the toolchain — exactly
the divergence #24 exists to close, one layer down. The stage now
copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
which ends in `go mod download`, so the separate invocation of that
is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves
above the bootstrap layer. It is the only edge making this stage
depend on the lint stage, so deleting it as redundant would end
fail-fast linting silently. Letting bootstrap install its own linter
here would have reintroduced the second toolchain and paid for a
from-source build of it. What makes the two stages provably one
toolchain rather than two that happen to agree is a new
`script/verify-linter-pin`, run in the build stage on the binary
that arrives from the lint stage, before bootstrap: it fails the
build naming both versions unless that binary is the version
`script/bootstrap` pins. Bootstrap's own check could not serve that
purpose — it reinstalls its pin from source and then verifies
whatever `PATH` resolves, so drift self-heals silently and a lint
stage image bumped on its own would lint at the new version while
`make check` ran at the old one, green. The linter version is pinned
in two independent places (the lint stage image digest and
- install the Docker build stage's prerequisites by running `script/bootstrap`
instead of `apk add --no-cache make` inline (2026-08-09, branch
`dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires
it, and the inline install left the build stage maintaining its own notion of
the toolchain — exactly the divergence #24 exists to close, one layer down.
The stage now copies `script/` plus `go.mod`/`go.sum` and runs
`script/bootstrap`, which ends in `go mod download`, so the separate
invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays,
and moves above the bootstrap layer. It is the only edge making this stage
depend on the lint stage, so deleting it as redundant would end fail-fast
linting silently. Letting bootstrap install its own linter here would have
reintroduced the second toolchain and paid for a from-source build of it. What
makes the two stages provably one toolchain rather than two that happen to
agree is a new `script/verify-linter-pin`, run in the build stage on the
binary that arrives from the lint stage, before bootstrap: it fails the build
naming both versions unless that binary is the version `script/bootstrap`
pins. Bootstrap's own check could not serve that purpose — it reinstalls its
pin from source and then verifies whatever `PATH` resolves, so drift
self-heals silently and a lint stage image bumped on its own would lint at the
new version while `make check` ran at the old one, green. The linter version
is pinned in two independent places (the lint stage image digest and
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
half-applied bump is now a build failure. The pin is read out of
`script/bootstrap`, which stays the single source of truth; a pin
that cannot be read is a hard failure, not a skip. The check needs
no `CHECK_EPOCH`: its only inputs are the copied binary and
`script/`, so Docker invalidates the layer exactly when a cached
result would stop being true, and it is documented with the other
entrypoints in the README. `$GOPATH/bin` joins `PATH` because
that is where bootstrap's `go install` lands and bootstrap verifies
its installs against what `PATH` resolves — nothing in the image is
shadowed by it, the directory does not exist until bootstrap runs.
Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and
`USER builder` still precede `make check`. Verified: the guard fails
the build with both versions named when the lint stage's linter is
faked to a different version, and an unmodified build still passes
it; bootstrap runs clean under Alpine's `sh` and its `apk` branch,
installing `git` and `make` and finding the copied
linter already at the pin; a second build served the bootstrap and
dependency layers `CACHED` while both gates ran with a fresh epoch;
a planted `unused` finding failed the build at the lint gate in
48.9s with the build stage's `make check` never starting; and the
suite run in the image as `--user 0:0` fails
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged
user is still load-bearing. That last check needs the Go test cache
disabled — the first attempt reported `ok ... (cached)` as root,
reusing the result the build-time run had left in the shared cache,
which would have read as a pass. Build wall time, on a shared host
running many concurrent builds and so noisy: 2m13s on an unchanged
tree, 2m17s and 4m29s for two builds after a source change, 5m14s
cold. Only the cold one breaches the policy ceiling, and not because
of this change — `chown -R builder:builder /src /home/builder` walks
the module cache and re-runs on every source change, and it alone
varied between 77s and 210s across those four builds, which is also
the whole spread in the totals. The same cold measurement against
`main` is 5m03s with a 209s `chown`. Filed as #43
- bust the Docker layer cache for the gate steps, so `script/cibuild`
and `script/docker` cannot report a green they did not earn
(2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts
were bare `docker build` invocations with no cache control, and the
`Dockerfile` copies the tree before running its gates, so on an
unchanged tree Docker served those layers from cache and the build
exited 0 having executed nothing. That is not hypothetical here —
every merge this repo has done is a non-fast-forward merge of an
undiverged branch, so each merge commit's tree is byte-identical to
the branch head's and each merge CI run was almost certainly a full
cache hit; and PR #31's reviewer found `make docker` returning
success as a 17-layer cache hit, catching it only by being
suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing
`--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break
it. `ARG` is scoped per stage and this `Dockerfile` has three gates
across two — `make fmt-check` and `make lint` in the lint stage,
`make check` in the build stage — so a single declaration would have
left one stage silently cacheable; it is declared in both. And
BuildKit hashes the expanded command, not the declaration, so a
declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN`
echoes the epoch, which also puts the value in the build log as
evidence the layer really ran. Placement is below the dependency
layers on purpose — a build that goes cold every time would be a
different bug, not a fix. Verified by running each script twice back
to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all
three gates executed on all four runs, each with a fresh epoch in
the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s
then 53.4s), and twelve steps were still served `CACHED` in the
steady state — both `go mod download`s, `apk add`, `adduser`, the
`chown`, every `go.mod`/`go.sum` and source copy, the linter copy
out of the lint stage, and the binary copy into the runtime stage.
The lint stage still gates the build stage: with a deliberate
`unused` finding planted in the tree, the build failed at
`make lint` in 36.1s and the build-stage `make check` never started.
The build stage also still drops to the unprivileged `builder` user
before `make check`, which the suite depends on rather than merely
prefers: forcing the same image to run the tests as root fails
`TestScanHardlinkRunFailsTogether`, because root reads straight
through the `chmod(0)` the test uses to prove hard links are read
once. This is the local fix only; propagating it to the canonical
templates is `prompts` #26
- check the installed golangci-lint version in `script/bootstrap`
instead of only its presence (2026-08-09, branch
`bootstrap-version-check`, closes #24): `missing golangci-lint` meant
any linter already on `PATH` satisfied the check, so the pin was never
consulted and the v2.12.2 bump from #3 was inert on every host that
already had one — this host ran v2.10.1 against a v2.12.2 pin,
`make check` went green, and `make docker` then rejected the same
commit with findings the local gate never saw. The version now lives
in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module
ref derived from it so a bump cannot half-apply; a
`golangci_lint_version` helper parses `golangci-lint --version`
(taking the field after the word `version` and tolerating an optional
leading `v`, which the module ref carries and the binary's output does
not), and any version that is not the pin — older, newer, absent or
unparseable — is reinstalled. The install is then verified against the
binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
first on `PATH`, so a wrong-version one sitting ahead of it — nix,
apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
stage makes — would swallow the install and leave the local gate
disagreeing with CI under an affirmative `bootstrap complete`.
Bootstrap now re-reads the effective version after installing and, on
a mismatch, prints both paths and both versions to stderr and exits
non-zero instead of claiming success; it does not reorder anyone's
`script/bootstrap`, which stays the single source of truth; a pin that cannot
be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its
only inputs are the copied binary and `script/`, so Docker invalidates the
layer exactly when a cached result would stop being true, and it is documented
with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because
that is where bootstrap's `go install` lands and bootstrap verifies its
installs against what `PATH` resolves — nothing in the image is shadowed by
it, the directory does not exist until bootstrap runs. Everything added sits
above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede
`make check`. Verified: the guard fails the build with both versions named
when the lint stage's linter is faked to a different version, and an
unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and
its `apk` branch, installing `git` and `make` and finding the copied linter
already at the pin; a second build served the bootstrap and dependency layers
`CACHED` while both gates ran with a fresh epoch; a planted `unused` finding
failed the build at the lint gate in 48.9s with the build stage's `make check`
never starting; and the suite run in the image as `--user 0:0` fails
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is
still load-bearing. That last check needs the Go test cache disabled — the
first attempt reported `ok ... (cached)` as root, reusing the result the
build-time run had left in the shared cache, which would have read as a pass.
Build wall time, on a shared host running many concurrent builds and so noisy:
2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source
change, 5m14s cold. Only the cold one breaches the policy ceiling, and not
because of this change — `chown -R builder:builder /src /home/builder` walks
the module cache and re-runs on every source change, and it alone varied
between 77s and 210s across those four builds, which is also the whole spread
in the totals. The same cold measurement against `main` is 5m03s with a 209s
`chown`. Filed as #43
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
`script/docker` cannot report a green they did not earn (2026-08-09, branch
`cibuild-cache-bust`, closes #32): both scripts were bare `docker build`
invocations with no cache control, and the `Dockerfile` copies the tree before
running its gates, so on an unchanged tree Docker served those layers from
cache and the build exited 0 having executed nothing. That is not hypothetical
here — every merge this repo has done is a non-fast-forward merge of an
undiverged branch, so each merge commit's tree is byte-identical to the branch
head's and each merge CI run was almost certainly a full cache hit; and PR
#31's reviewer found `make docker` returning success as a 17-layer cache hit,
catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the
scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or
break it. `ARG` is scoped per stage and this `Dockerfile` has three gates
across two — `make fmt-check` and `make lint` in the lint stage, `make check`
in the build stage — so a single declaration would have left one stage
silently cacheable; it is declared in both. And BuildKit hashes the expanded
command, not the declaration, so a declared-but-unreferenced `ARG` invalidates
nothing: each gate `RUN` echoes the epoch, which also puts the value in the
build log as evidence the layer really ran. Placement is below the dependency
layers on purpose — a build that goes cold every time would be a different
bug, not a fix. Verified by running each script twice back to back on an
unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on
all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then
61.1s; `script/docker` 61.1s then 53.4s), and twelve steps were still served
`CACHED` in the steady state — both `go mod download`s, `apk add`, `adduser`,
the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy out of
the lint stage, and the binary copy into the runtime stage. The lint stage
still gates the build stage: with a deliberate `unused` finding planted in the
tree, the build failed at `make lint` in 36.1s and the build-stage
`make check` never started. The build stage also still drops to the
unprivileged `builder` user before `make check`, which the suite depends on
rather than merely prefers: forcing the same image to run the tests as root
fails `TestScanHardlinkRunFailsTogether`, because root reads straight through
the `chmod(0)` the test uses to prove hard links are read once. This is the
local fix only; propagating it to the canonical templates is `prompts` #26
- check the installed golangci-lint version in `script/bootstrap` instead of
only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24):
`missing golangci-lint` meant any linter already on `PATH` satisfied the
check, so the pin was never consulted and the v2.12.2 bump from #3 was inert
on every host that already had one — this host ran v2.10.1 against a v2.12.2
pin, `make check` went green, and `make docker` then rejected the same commit
with findings the local gate never saw. The version now lives in one place,
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a
bump cannot half-apply; a `golangci_lint_version` helper parses
`golangci-lint --version` (taking the field after the word `version` and
tolerating an optional leading `v`, which the module ref carries and the
binary's output does not), and any version that is not the pin — older, newer,
absent or unparseable — is reinstalled. The install is then verified against
the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on
`PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or
the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow
the install and leave the local gate disagreeing with CI under an affirmative
`bootstrap complete`. Bootstrap now re-reads the effective version after
installing and, on a mismatch, prints both paths and both versions to stderr
and exits non-zero instead of claiming success; it does not reorder anyone's
`PATH` or delete their binary. The `--version` call keeps its stderr
connected, so a present-but-broken binary says why rather than
reinstalling forever in silence, and is bounded by `timeout(1)` where
that exists, so a wedged binary cannot hang bootstrap. `git`, `make`
and `go` keep their presence-only checks and now say why in a
comment: they are host package-manager tools the repo deliberately
does not pin, with `go.mod` governing the language version and the
digest-pinned images covering reproducible builds. Verified on this
host by bootstrapping from v2.10.1 to v2.12.2 and running it again to
a no-op, plus stub runs of the real script under `dash` covering a
thirteen-input parse matrix (absent, older, newer, host-style,
image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor
binary, `(devel)`, trailing `version`), a shadowed install that must
exit non-zero, an install destination not on `PATH` at all, `GOBIN`
set, and a wedged binary that must hit the timeout; `make check` and
`make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
findings on `main`
connected, so a present-but-broken binary says why rather than reinstalling
forever in silence, and is bounded by `timeout(1)` where that exists, so a
wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their
presence-only checks and now say why in a comment: they are host
package-manager tools the repo deliberately does not pin, with `go.mod`
governing the language version and the digest-pinned images covering
reproducible builds. Verified on this host by bootstrapping from v2.10.1 to
v2.12.2 and running it again to a no-op, plus stub runs of the real script
under `dash` covering a thirteen-input parse matrix (absent, older, newer,
host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit,
impostor binary, `(devel)`, trailing `version`), a shadowed install that must
exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a
wedged binary that must hit the timeout; `make check` and `make lint` are
clean at v2.12.2, so v2.10.1 was not hiding any findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the
moment `recordRun` failed and abandon the pool — the feeder parked
forever on a full `jobs` channel and every worker on a full
`results` channel. That only stopped being invisible when #4 landed
and `runScan` began unwinding instead of calling `os.Exit`. The
pool is now an owned, context-aware `hashPool`: every blocking send
in the feeder and the workers selects on `ctx.Done()`, `jobs` is
closed on every path out, and `hashPhase` defers `pool.stop()`,
which cancels and then drains `results` until the last goroutine
has exited — draining is what frees a worker already parked on a
send. `ctx` is threaded from `cmd.Context()` through `runScan`,
`syncScan`, both worker pools and the whole database layer (it is
the first parameter everywhere), so #5 can hand this path a signal
and needs to add nothing else. The walk pool never leaked, because
`walkPhase` always drains its events to close, but it has the same
unbounded-send shape and #5 will give it an early return, so it
gets the same treatment plus a `ctx.Err()` guard after the walk: a
cancelled walk yields a partial size census, and every file it never
reached looks vanished to the update phase. That phase's own
`BeginTx` fails on the same cancelled context before deleting
anything, so the guard is defence in depth rather than the only
barrier — but it is the one that survives #5 deciding an interrupted
scan may commit what it has. Tests drive `run(scan)` against a
database whose insert trigger aborts, and assert both that the scan
fails instead of hanging and that `runtime.NumGoroutine()` polls
back to its pre-scan baseline; a second set cancels a scan part-way
through the walk — deterministically, by counting the scan's own
consultations of `ctx.Done()` rather than racing a timer — and
asserts that it stops at the guard holding a partial census and a
still-populated record index, with every record intact. The
remaining cancellation branches of both pools are covered by direct
tests of `sendEvent`, the walk workers, `dispatchDirs`,
`feedHashJobs`, `hashWorker` and `hashPhase`
- guarantee the database is closed on every fatal exit path
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
it the SQLite WAL checkpoint — now actually runs when a subcommand
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
`resolveRoots` return errors instead. The single exit point is `run`
in `main.go`: it maps a `fatalError` (anything a subcommand
returned) to exit 1 and cobra's own argument and flag errors to exit
2, which keeps a runtime failure from being reported as a usage
error or printing the usage text. New `main_test.go` drives the CLI
in-process and asserts the exit codes from README §Error handling
plus the stdout/stderr split, including that a fatal error raised
after the database is open leaves no `-wal`/`-shm` sidecar behind
for `scan`, `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
closes #3): bumped the pinned linter in the `Dockerfile` lint
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
`.golangci.yml` with the canonical file — the linter settings
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment
`recordRun` failed and abandon the pool — the feeder parked forever on a full
`jobs` channel and every worker on a full `results` channel. That only stopped
being invisible when #4 landed and `runScan` began unwinding instead of
calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every
blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is
closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels
and then drains `results` until the last goroutine has exited — draining is
what frees a worker already parked on a send. `ctx` is threaded from
`cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole
database layer (it is the first parameter everywhere), so #5 can hand this
path a signal and needs to add nothing else. The walk pool never leaked,
because `walkPhase` always drains its events to close, but it has the same
unbounded-send shape and #5 will give it an early return, so it gets the same
treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a
partial size census, and every file it never reached looks vanished to the
update phase. That phase's own `BeginTx` fails on the same cancelled context
before deleting anything, so the guard is defence in depth rather than the
only barrier — but it is the one that survives #5 deciding an interrupted scan
may commit what it has. Tests drive `run(scan)` against a database whose
insert trigger aborts, and assert both that the scan fails instead of hanging
and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a
second set cancels a scan part-way through the walk — deterministically, by
counting the scan's own consultations of `ctx.Done()` rather than racing a
timer — and asserts that it stops at the guard holding a partial census and a
still-populated record index, with every record intact. The remaining
cancellation branches of both pools are covered by direct tests of
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
and `hashPhase`
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
`db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so
the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now
actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`,
`loadRecords` and `resolveRoots` return errors instead. The single exit point
is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned)
to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a
runtime failure from being reported as a usage error or printing the usage
text. New `main_test.go` drives the CLI in-process and asserts the exit codes
from README §Error handling plus the stdout/stderr split, including that a
fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar
behind for `scan`, `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
`golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter
in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2,
and replaced `.golangci.yml` with the canonical file — the linter settings
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
`linters.settings` per the v2 schema, so they are actually
applied; no new lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos (2026-07-26, commit
`3abeacf`, closes #1): all 12 `script/` entrypoints exist
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`) and every Makefile target is now a thin shim
over them, matching the other managed repos
`linters.settings` per the v2 schema, so they are actually applied; no new
lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12
`script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`,
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`install-precommit`) and every Makefile target is now a thin shim over them,
matching the other managed repos
- make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes`
(previously it ran `check` plus `build`); `make build` remains as
an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
branch `scan-wide-phases`): all operands seed the shared walk pool
and every pass runs once over the whole scan, so totals and ETAs
are scan-global; the per-operand walk/hash/update cycles and their
stderr announcements are gone; the update pass commits in batched
transactions — the filesystem is authoritative and the database an
eventually-consistent reflection, so scan-level atomicity is not
required
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran
`check` plus `build`); `make build` remains as an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
`scan-wide-phases`): all operands seed the shared walk pool and every pass
runs once over the whole scan, so totals and ETAs are scan-global; the
per-operand walk/hash/update cycles and their stderr announcements are gone;
the update pass commits in batched transactions — the filesystem is
authoritative and the database an eventually-consistent reflection, so
scan-level atomicity is not required
- split the stat pass back out of the walk (2026-07-24, branch
`parallel-phases`): phases are strictly sequential again — walk,
stat, hash, update per operand — with parallelism only inside each
phase; the walk enumerates paths with per-directory workers and the
stat pass lstats them with per-file workers, restoring the exact
total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24,
branch `scan-operand-progress`): with per-operand walk/hash/update
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
that looked like the whole run's — an operator watching operand 3 of
14 hash 300k files concluded 20M files were being skipped
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
was a single goroutine and took hours at ~20M files on a busy pool
(observed: 22M files in 4h on a ZFS server); it is now a
per-directory worker-pool traversal that records size/mtime during
the walk (folding away the separate stat pass, halving metadata
I/O), and each `PATH` operand commits in its own transaction so an
interrupted scan keeps completed operands
`parallel-phases`): phases are strictly sequential again — walk, stat, hash,
update per operand — with parallelism only inside each phase; the walk
enumerates paths with per-directory workers and the stat pass lstats them with
per-file workers, restoring the exact total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24, branch
`scan-operand-progress`): with per-operand walk/hash/update cycles, a
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
whole run's — an operator watching operand 3 of 14 hash 300k files concluded
20M files were being skipped
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
4h on a ZFS server); it is now a per-directory worker-pool traversal that
records size/mtime during the walk (folding away the separate stat pass,
halving metadata I/O), and each `PATH` operand commits in its own transaction
so an interrupted scan keeps completed operands
- persistent scan database (2026-07-24, branch `persistent-database`):
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
Go, cgo stays disabled) keyed by absolute path that survives between
runs — a rescan hashes only new or changed files (by mtime/size),
deletes records for files vanished from under the scanned operands,
and leaves records outside them untouched, so `scan` can be cronned
daily; `report` and `trees` read the database (no positional
arguments) instead of a scan stream. Database at
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
WAL journaling plus a single-transaction update keep a report run
during a scan safe
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
`v0.0.1`, and push `main` plus tags (2026-07-23)
- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
disabled) keyed by absolute path that survives between runs — a rescan hashes
only new or changed files (by mtime/size), deletes records for files vanished
from under the scanned operands, and leaves records outside them untouched, so
`scan` can be cronned daily; `report` and `trees` read the database (no
positional arguments) instead of a scan stream. Database at
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
journaling plus a single-transaction update keep a report run during a scan
safe
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
`PATH...` operands via cobra flags replacing the `/srv` `-root`
default; new `-x`/`--one-file-system` flag (GNU convention) to stop
at filesystem boundaries, which are crossed by default
`PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
`-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
boundaries, which are crossed by default
- bring the repo into full policy compliance (2026-07-23, branch
`repo-policy-compliance`; checklist below)
- `git init` with README-only first commit; code baseline committed on
`main` (2026-07-22)
- `git init` with README-only first commit; code baseline committed on `main`
(2026-07-22)
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
# Future Steps
- possible later features (explicitly out of scope per README):
full-content verification of candidates, removal-script helpers
- possible later features (explicitly out of scope per README): full-content
verification of candidates, removal-script helpers
# Repo Policy Compliance
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
standalone formatting commit is needed.
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
formatting commit is needed.
- [x] `.gitignore` missing — the compiled `sfdupes` binary and
`files.dat` sit untracked in the tree; needs OS/editor/Go
artifacts plus secrets patterns
- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
- [x] `.editorconfig` missing
- [x] `LICENSE` missing and README has no License section (MIT assumed
from house convention — user to confirm)
- [x] `LICENSE` missing and README has no License section (MIT assumed from
house convention — user to confirm)
- [x] `REPO_POLICIES.md` missing from repo root
- [x] `.golangci.yml` missing (install canonical copy); code must then
pass `make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
`fmt-check`, `docker`, `hooks`; `check` currently depends on
`build`, which writes the binary (`make check` must not modify
files)
- [x] no tests — `go test ./...` has nothing to run; policy requires
real tests with a 30-second timeout and the conditional `-v`
rerun pattern (suite covers parsing, grouping, digests,
suppression, hashing, and the scan pipeline; 64% coverage)
- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
fail-fast lint stage, build stage running `make check`
- [x] `.golangci.yml` missing (install canonical copy); code must then pass
`make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
`docker`, `hooks`; `check` currently depends on `build`, which writes the
binary (`make check` must not modify files)
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
with a 30-second timeout and the conditional `-v` rerun pattern (suite
covers parsing, grouping, digests, suppression, hashing, and the scan
pipeline; 64% coverage)
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
lint stage, build stage running `make check`
- [x] `.dockerignore` missing
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
checkout action pinned by commit SHA)
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
action pinned by commit SHA)
- [x] README lacks required sections: Description first line
(name/purpose/category/license/author), Getting Started,
Rationale, TODO, License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now
that the repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target
exists)
(name/purpose/category/license/author), Getting Started, Rationale, TODO,
License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now that the
repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target exists)
Accepted divergences (no action):
- flat single-package layout with `.go` files in the repo root — fine
for a small single-binary tool per the Go styleguide; the tracker
audit agrees
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
(pure-Go builds) and the race detector requires cgo
- flat single-package layout with `.go` files in the repo root — fine for a
small single-binary tool per the Go styleguide; the tracker audit agrees
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go
builds) and the race detector requires cgo
+5
View File
@@ -0,0 +1,5 @@
{
"devDependencies": {
"prettier": "3.8.1"
}
}
+31
View File
@@ -11,6 +11,12 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# yarn provides prettier, which formats Markdown. yarn is a tool, like
# node/git/make/go below; the reference that governs formatting output is
# prettier, pinned by yarn.lock's integrity hash and installed by
# `yarn install --frozen-lockfile`.
YARN_VERSION="1.22.22"
PKGMGR=""
SUDO=""
APT_UPDATED=""
@@ -58,6 +64,21 @@ missing() {
! command -v "$1" >/dev/null 2>&1
}
ensure_node() {
if ! missing node; then return 0; fi
pkg_install nodejs nodejs node nodejs
}
ensure_yarn() {
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable >/dev/null 2>&1 || true
corepack prepare "yarn@$YARN_VERSION" --activate
else
pkg_install yarn yarn yarn yarn
fi
}
main() {
cd "$ROOT"
@@ -71,6 +92,16 @@ main() {
if missing make; then pkg_install gnumake make make make; fi
if missing go; then pkg_install go golang go go; fi
# node runs prettier and is an unpinned host tool for the same reason
# git/make/go are: it comes from the host package manager, whatever
# version it ships. It is not installed via nvm the way the canonical
# template does, because nvm's prebuilt node is glibc-linked and does
# not run on this repo's musl/Alpine build image. prettier — the tool
# whose version affects formatting output — is pinned by yarn.lock.
ensure_node
ensure_yarn
yarn install --frozen-lockfile
# Linting runs via docker only (script/lint), so docker is a lint
# prerequisite rather than something bootstrap installs. Warn, do
# not fail: everything except `make lint` — and, through it,
+12 -1
View File
@@ -1,12 +1,23 @@
#!/bin/sh
# script/fmt: format all files (writes).
# script/fmt: format all files (writes). gofmt for Go, prettier for
# Markdown. prettier is the pinned devDependency in package.json/
# yarn.lock; script/bootstrap installs it (see run_prettier).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
run_prettier() {
if ! command -v yarn >/dev/null 2>&1; then
echo "fmt: yarn not found; run script/bootstrap first" >&2
exit 1
fi
yarn run prettier "$@"
}
main() {
cd "$ROOT"
gofmt -s -w .
run_prettier --write '**/*.md' --tab-width 4 --prose-wrap always
}
main "$@"
+21 -2
View File
@@ -1,18 +1,37 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing.
# script/fmt: gofmt for Go, prettier for Markdown. Both run every time
# and each reports independently, so a failure names which formatter is
# unhappy; the script exits non-zero if either found unformatted files.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
run_prettier() {
if ! command -v yarn >/dev/null 2>&1; then
echo "fmt-check: yarn not found; run script/bootstrap first" >&2
exit 1
fi
yarn run prettier "$@"
}
main() {
cd "$ROOT"
rc=0
files="$(gofmt -s -l .)"
if [ -n "$files" ]; then
echo "gofmt: files not formatted:" >&2
echo "$files" >&2
exit 1
rc=1
fi
if ! run_prettier --check '**/*.md' --tab-width 4 --prose-wrap always; then
echo "prettier: Markdown not formatted; run make fmt" >&2
rc=1
fi
exit "$rc"
}
main "$@"
+8
View File
@@ -0,0 +1,8 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
prettier@3.8.1:
version "3.8.1"
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==