Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2479ebe7cb |
@@ -2,8 +2,6 @@
|
||||
.claude
|
||||
.DS_Store
|
||||
sfdupes
|
||||
files.dat
|
||||
node_modules
|
||||
*.log
|
||||
*.out
|
||||
*.test
|
||||
|
||||
@@ -27,7 +27,6 @@ node_modules/
|
||||
*.log
|
||||
|
||||
# Local scan data
|
||||
files.dat
|
||||
*.sqlite
|
||||
*.sqlite-shm
|
||||
*.sqlite-wal
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
node_modules/
|
||||
yarn.lock
|
||||
@@ -1,4 +0,0 @@
|
||||
{
|
||||
"tabWidth": 4,
|
||||
"proseWrap": "always"
|
||||
}
|
||||
+6
-11
@@ -27,12 +27,9 @@ ARG CHECK_EPOCH
|
||||
# target now runs `docker build -f Dockerfile.lint`, and a docker build
|
||||
# cannot run a docker build: routing the gate through make would mean
|
||||
# nesting docker inside this image. Same reason `make check` is gone
|
||||
# from the build stage below.
|
||||
#
|
||||
# `make fmt-check` is not run in this stage: it now also runs prettier
|
||||
# over Markdown, and this golangci-lint image has no node. The gate runs
|
||||
# in the build stage below, where script/bootstrap installs node and
|
||||
# prettier.
|
||||
# from the build stage below. `make fmt-check` stays as it is — it is a
|
||||
# gate, not the aggregate, and it shells out to nothing.
|
||||
RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
|
||||
|
||||
# The FROM above and the one in Dockerfile.lint pin the same linter
|
||||
# twice, and nothing else keeps them in sync; this fails the build when
|
||||
@@ -80,12 +77,10 @@ COPY --from=lint /src/go.sum /dev/null
|
||||
# rather than duplicating the installs inline. Only script/ and the
|
||||
# dependency manifests are copied first, nothing else, so this layer
|
||||
# stays cached until the scripts or the dependencies change — bootstrap
|
||||
# runs `go mod download` and `yarn install`, which is why there is no
|
||||
# separate invocation of either here. The JS manifests (package.json,
|
||||
# yarn.lock) are copied too so the yarn install layer caches alongside
|
||||
# the Go one.
|
||||
# ends in `go mod download`, which is why there is no separate
|
||||
# invocation of it here.
|
||||
COPY script/ script/
|
||||
COPY go.mod go.sum package.json yarn.lock ./
|
||||
COPY go.mod go.sum ./
|
||||
RUN script/bootstrap
|
||||
|
||||
COPY . .
|
||||
|
||||
@@ -46,4 +46,4 @@ hooks:
|
||||
@script/install-precommit
|
||||
|
||||
clean:
|
||||
rm -f $(BINARY) files.dat
|
||||
rm -f $(BINARY)
|
||||
|
||||
@@ -2,18 +2,19 @@
|
||||
|
||||
## Description
|
||||
|
||||
`sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that
|
||||
quickly identifies _candidate_ duplicate files — and, ultimately, entire
|
||||
duplicate directory trees — across very large filesystems without reading full
|
||||
file contents. Files are considered duplicates when they have identical size,
|
||||
identical SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
|
||||
1024 bytes. This is a strong candidate signal, not proof of identical content
|
||||
(the middle of the file is never read); the intended use is finding duplicate
|
||||
downloads and duplicated directory trees on multi-terabyte ZFS servers where
|
||||
reading every byte is prohibitively expensive. `scan` maintains a persistent
|
||||
SQLite database of file signatures that survives between runs, so it can be run
|
||||
from cron and the reports can be generated at any time from the most recent
|
||||
scan.
|
||||
`sfdupes` is an MIT-licensed Go CLI tool by
|
||||
[@sneak](https://sneak.berlin) that quickly identifies *candidate*
|
||||
duplicate files — and, ultimately, entire duplicate directory trees —
|
||||
across very large filesystems without reading full file contents. Files
|
||||
are considered duplicates when they have identical size, identical
|
||||
SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
|
||||
1024 bytes. This is a strong candidate signal, not proof of identical
|
||||
content (the middle of the file is never read); the intended use is
|
||||
finding duplicate downloads and duplicated directory trees on
|
||||
multi-terabyte ZFS servers where reading every byte is prohibitively
|
||||
expensive. `scan` maintains a persistent SQLite database of file
|
||||
signatures that survives between runs, so it can be run from cron and
|
||||
the reports can be generated at any time from the most recent scan.
|
||||
|
||||
This README is the complete and authoritative specification.
|
||||
|
||||
@@ -27,81 +28,91 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
||||
./sfdupes trees > dupetrees.tsv
|
||||
```
|
||||
|
||||
`scan` walks one or more filesystem trees and maintains one database record per
|
||||
regular file (path, size, mtime, head hash, tail hash). The database persists
|
||||
between runs; a rescan only hashes files that are new or changed, and removes
|
||||
records for files that no longer exist. `report` reads the database and prints
|
||||
the file-level duplicates report. `trees` reads the same database and prints the
|
||||
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
|
||||
with no `PATH` operand — prints a usage message and exits 2.
|
||||
`scan` walks one or more filesystem trees and maintains one database
|
||||
record per regular file (path, size, mtime, head hash, tail hash). The
|
||||
database persists between runs; a rescan only hashes files that are new
|
||||
or changed, and removes records for files that no longer exist.
|
||||
`report` reads the database and prints the file-level duplicates
|
||||
report. `trees` reads the same database and prints the duplicate-tree
|
||||
report. A missing/invalid subcommand — or a `scan` invocation with no
|
||||
`PATH` operand — prints a usage message and exits 2.
|
||||
|
||||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
||||
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
||||
cron job, with the reporting commands run interactively whenever needed; their
|
||||
results are as fresh as the last completed scan.
|
||||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed
|
||||
anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a
|
||||
daily `sfdupes scan` cron job, with the reporting commands run
|
||||
interactively whenever needed; their results are as fresh as the last
|
||||
completed scan.
|
||||
|
||||
## Rationale
|
||||
|
||||
Duplicate finders that hash entire files do not scale to the target environment:
|
||||
~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under
|
||||
resilver). Reading at most 2 KiB per file — and only from files whose size at
|
||||
least one other file shares, since a size-unique file cannot be a duplicate —
|
||||
makes a full-filesystem sweep tractable, and the signatures are kept in a
|
||||
persistent database, so the expensive filesystem pass is incremental: a rescan
|
||||
re-hashes only files whose recorded mtime or size changed, and all analysis
|
||||
happens offline from the database alone. The end goal is not individual files
|
||||
but whole duplicated trees — duplicate extractions, duplicate downloads, copied
|
||||
project trees — which an operator can consider removing as a unit.
|
||||
Duplicate finders that hash entire files do not scale to the target
|
||||
environment: ~10 million files and ~150 TB on possibly slow or busy
|
||||
disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
|
||||
only from files whose size at least one other file shares, since a
|
||||
size-unique file cannot be a duplicate — makes a full-filesystem sweep
|
||||
tractable, and the signatures are kept in a persistent database, so
|
||||
the expensive filesystem pass is incremental: a rescan re-hashes only
|
||||
files whose recorded mtime or size changed, and all analysis happens
|
||||
offline from the database alone. The end goal is
|
||||
not individual files but whole duplicated trees — duplicate
|
||||
extractions, duplicate downloads, copied project trees — which an
|
||||
operator can consider removing as a unit.
|
||||
|
||||
## Design
|
||||
|
||||
Goals, in order:
|
||||
|
||||
1. **Find whole duplicate trees, not just files.** The end goal is to identify
|
||||
places where the exact same set of files and directories exists at two or
|
||||
more paths (duplicate extractions, duplicate downloads, copied project
|
||||
trees), so the operator can consider removing an entire subtree at once.
|
||||
File-level duplicate detection is the foundation; tree-level detection is
|
||||
built on top of it.
|
||||
2. **Never read full file contents.** At most 2 KiB is read per file (first and
|
||||
last 1024 bytes), and only files whose size at least one other file shares
|
||||
are read at all — a size-unique file cannot be a duplicate. Scale target:
|
||||
tens of millions of files, ~150 TB filesystem, possibly slow or busy disks
|
||||
(ZFS pool under resilver). Holding one small record (path, size, mtime) per
|
||||
file in memory during a scan is acceptable; holding every file's hashes is
|
||||
not (they stay in the database).
|
||||
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
|
||||
maintains a persistent database; an unchanged file is never read again on a
|
||||
rescan. All analysis (`report`, `trees`) works from the database alone and
|
||||
must never touch the scanned filesystem again. `scan` is designed to be
|
||||
cronned; the reports run at any time against the last completed scan.
|
||||
4. **Clean stream separation.** Everything on stdout is machine-readable data.
|
||||
All progress, warnings, and summaries go to stderr. Never mix them.
|
||||
1. **Find whole duplicate trees, not just files.** The end goal is to
|
||||
identify places where the exact same set of files and directories
|
||||
exists at two or more paths (duplicate extractions, duplicate
|
||||
downloads, copied project trees), so the operator can consider
|
||||
removing an entire subtree at once. File-level duplicate detection is
|
||||
the foundation; tree-level detection is built on top of it.
|
||||
2. **Never read full file contents.** At most 2 KiB is read per file
|
||||
(first and last 1024 bytes), and only files whose size at least
|
||||
one other file shares are read at all — a size-unique file cannot
|
||||
be a duplicate. Scale target: tens of millions of files, ~150 TB
|
||||
filesystem, possibly slow or busy disks (ZFS pool under resilver).
|
||||
Holding one small record (path, size, mtime) per file in memory
|
||||
during a scan is acceptable; holding every file's hashes is not
|
||||
(they stay in the database).
|
||||
3. **Scan incrementally, analyze offline.** The expensive filesystem
|
||||
scan maintains a persistent database; an unchanged file is never
|
||||
read again on a rescan. All analysis (`report`, `trees`) works from
|
||||
the database alone and must never touch the scanned filesystem
|
||||
again. `scan` is designed to be cronned; the reports run at any
|
||||
time against the last completed scan.
|
||||
4. **Clean stream separation.** Everything on stdout is machine-readable
|
||||
data. All progress, warnings, and summaries go to stderr. Never mix
|
||||
them.
|
||||
|
||||
### Constraints
|
||||
|
||||
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`.
|
||||
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, **one
|
||||
progress-bar library** (`github.com/schollz/progressbar/v3`), and **one SQLite
|
||||
driver** (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
|
||||
`github.com/spf13/viper` is permitted if configuration-file support is ever
|
||||
needed, but is not currently used. No other third-party deps.
|
||||
- Cross-compilation is not a concern. Builds run with cgo disabled (the
|
||||
`Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
|
||||
- Analysis modes (`report`, `trees`) must be deterministic: identical database
|
||||
contents, identical output, regardless of the order in which records were
|
||||
inserted.
|
||||
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
|
||||
`sfdupes`.
|
||||
- Dependencies: standard library, `github.com/spf13/cobra` for the
|
||||
CLI, **one progress-bar library**
|
||||
(`github.com/schollz/progressbar/v3`), and **one SQLite driver**
|
||||
(`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
|
||||
`github.com/spf13/viper` is permitted if configuration-file support
|
||||
is ever needed, but is not currently used. No other third-party
|
||||
deps.
|
||||
- Cross-compilation is not a concern. Builds run with cgo disabled (the
|
||||
`Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
|
||||
- Analysis modes (`report`, `trees`) must be deterministic: identical
|
||||
database contents, identical output, regardless of the order in
|
||||
which records were inserted.
|
||||
|
||||
### Subcommands
|
||||
|
||||
Three subcommands, all implemented:
|
||||
|
||||
1. `scan` — walk the filesystem and synchronize the database: one signature
|
||||
record per regular file.
|
||||
1. `scan` — walk the filesystem and synchronize the database: one
|
||||
signature record per regular file.
|
||||
2. `report` — file-level duplicate report from the database.
|
||||
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
|
||||
from the database records, compute a Merkle-style digest per directory, and
|
||||
report maximal groups of identical trees.
|
||||
3. `trees` — tree-level duplicate report: reconstruct the directory
|
||||
hierarchy from the database records, compute a Merkle-style digest
|
||||
per directory, and report maximal groups of identical trees.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] PATH...
|
||||
@@ -113,22 +124,25 @@ sfdupes trees > dupetrees.tsv
|
||||
|
||||
All three subcommands operate on a single SQLite database file:
|
||||
|
||||
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
|
||||
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
|
||||
command-line flag.
|
||||
- `scan` creates the database (and its parent directory) on first use. `report`
|
||||
and `trees` require an existing database; a missing database file is a fatal
|
||||
error (exit 1) telling the user to run `scan` first.
|
||||
- The database uses WAL journal mode and a busy timeout, so running a report
|
||||
while a cron `scan` is in progress is safe. The filesystem is authoritative;
|
||||
the database is an eventually-consistent reflection of it. Hashed records are
|
||||
committed in batched transactions while the scan is still running (keeping the
|
||||
WAL small and letting concurrent reports observe progress), so a report may
|
||||
see a scan's changes partially applied, and a scan that dies partway leaves a
|
||||
valid database holding everything hashed so far; the next scan skips those
|
||||
records and converges toward the filesystem.
|
||||
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
|
||||
with any other version is a fatal error):
|
||||
- Location: the value of the `SFDUPES_DATABASE` environment variable
|
||||
when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`.
|
||||
There is no command-line flag.
|
||||
- `scan` creates the database (and its parent directory) on first
|
||||
use. `report` and `trees` require an existing database; a missing
|
||||
database file is a fatal error (exit 1) telling the user to run
|
||||
`scan` first.
|
||||
- The database uses WAL journal mode and a busy timeout, so running a
|
||||
report while a cron `scan` is in progress is safe. The filesystem
|
||||
is authoritative; the database is an eventually-consistent
|
||||
reflection of it. Hashed records are committed in batched
|
||||
transactions while the scan is still running (keeping the WAL
|
||||
small and letting concurrent reports observe progress), so a
|
||||
report may see a scan's changes partially applied, and a scan
|
||||
that dies partway leaves a valid database holding everything
|
||||
hashed so far; the next scan skips those records and converges
|
||||
toward the filesystem.
|
||||
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
|
||||
database with any other version is a fatal error):
|
||||
|
||||
```sql
|
||||
CREATE TABLE files (
|
||||
@@ -140,147 +154,167 @@ All three subcommands operate on a single SQLite database file:
|
||||
) WITHOUT ROWID;
|
||||
```
|
||||
|
||||
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
|
||||
UTF-8. `mtime` is used only for change detection; it is not part of the
|
||||
duplicate key. `head` and `tail` are empty strings when the file has never
|
||||
been hashed because its size was unique as of the last scan that covered it;
|
||||
such records still define the file for tree reconstruction but never
|
||||
participate in duplicate groups.
|
||||
Paths are stored as BLOBs because Unix paths are raw bytes, not
|
||||
guaranteed UTF-8. `mtime` is used only for change detection; it is
|
||||
not part of the duplicate key. `head` and `tail` are empty strings
|
||||
when the file has never been hashed because its size was unique as
|
||||
of the last scan that covered it; such records still define the
|
||||
file for tree reconstruction but never participate in duplicate
|
||||
groups.
|
||||
|
||||
### `scan` mode
|
||||
|
||||
`scan` requires one or more `PATH` operands naming the trees to scan. There is
|
||||
no default path; invoking `scan` with no operand is a usage error (usage message
|
||||
on stderr, exit 2). An operand may be a directory or a regular file; an operand
|
||||
that does not exist is a fatal error (exit 1). Because database records persist
|
||||
between runs and are keyed by absolute path, each operand is resolved to an
|
||||
absolute, lexically cleaned path (symlinks are not resolved) before walking, so
|
||||
results do not depend on the working directory. All operands belong to a single
|
||||
scan and are enumerated concurrently: every operand seeds the shared walk worker
|
||||
pool. Overlapping operands are harmless — an operand that duplicates another or
|
||||
lies under another is dropped before walking, so every file is reached exactly
|
||||
once and produces one database record.
|
||||
`scan` requires one or more `PATH` operands naming the trees to scan.
|
||||
There is no default path; invoking `scan` with no operand is a usage
|
||||
error (usage message on stderr, exit 2). An operand may be a directory
|
||||
or a regular file; an operand that does not exist is a fatal error
|
||||
(exit 1). Because database records persist between runs and are keyed
|
||||
by absolute path, each operand is resolved to an absolute, lexically
|
||||
cleaned path (symlinks are not resolved) before walking, so results do
|
||||
not depend on the working directory. All operands belong to a single
|
||||
scan and are enumerated concurrently: every operand seeds the shared
|
||||
walk worker pool. Overlapping operands are harmless — an operand that
|
||||
duplicates another or lies under another is dropped before walking,
|
||||
so every file is reached exactly once and produces one database
|
||||
record.
|
||||
|
||||
`scan` synchronizes the database with the filesystem state under the scanned
|
||||
operands:
|
||||
`scan` synchronizes the database with the filesystem state under the
|
||||
scanned operands:
|
||||
|
||||
- Only a file whose size at least one other file shares is ever read: a
|
||||
size-unique file cannot be a duplicate, so it is recorded without hashes
|
||||
(`head` and `tail` empty). The size census covers every file walked this scan
|
||||
plus every database record outside the scanned operands, so a possible
|
||||
duplicate of a separately scanned tree is still recognized.
|
||||
- A file not yet in the database is inserted: hashed when its size is shared,
|
||||
without hashes otherwise.
|
||||
- A file already in the database is **skipped without reading its contents**
|
||||
when its lstat size equals the recorded size and its lstat mtime is not newer
|
||||
than the recorded mtime. This is what makes a daily rescan cheap. Exception:
|
||||
an unchanged file whose record lacks hashes is hashed — and its record updated
|
||||
— once its size becomes shared, so hashing deferred by size-uniqueness happens
|
||||
as soon as it could matter.
|
||||
- A file whose mtime is newer than recorded, or whose size differs, is processed
|
||||
as if new: re-hashed, or recorded without hashes, per the shared-size rule.
|
||||
- A database record whose path lies under one of the scanned operands but was
|
||||
not successfully processed this run is deleted. This removes records for
|
||||
deleted files. It also removes records for paths that failed to stat or hash
|
||||
this run: the database only ever contains signatures verified by the most
|
||||
recent scan that covered them (a subsequent successful scan re-adds such
|
||||
files).
|
||||
- Database records outside the scanned operands are untouched, so disjoint trees
|
||||
can be scanned on different schedules into the same database.
|
||||
- Only a file whose size at least one other file shares is ever
|
||||
read: a size-unique file cannot be a duplicate, so it is recorded
|
||||
without hashes (`head` and `tail` empty). The size census covers
|
||||
every file walked this scan plus every database record outside
|
||||
the scanned operands, so a possible duplicate of a separately
|
||||
scanned tree is still recognized.
|
||||
- A file not yet in the database is inserted: hashed when its size
|
||||
is shared, without hashes otherwise.
|
||||
- A file already in the database is **skipped without reading its
|
||||
contents** when its lstat size equals the recorded size and its
|
||||
lstat mtime is not newer than the recorded mtime. This is what
|
||||
makes a daily rescan cheap. Exception: an unchanged file whose
|
||||
record lacks hashes is hashed — and its record updated — once its
|
||||
size becomes shared, so hashing deferred by size-uniqueness
|
||||
happens as soon as it could matter.
|
||||
- A file whose mtime is newer than recorded, or whose size differs,
|
||||
is processed as if new: re-hashed, or recorded without hashes,
|
||||
per the shared-size rule.
|
||||
- A database record whose path lies under one of the scanned operands
|
||||
but was not successfully processed this run is deleted. This
|
||||
removes records for deleted files. It also removes records for
|
||||
paths that failed to stat or hash this run: the database only ever
|
||||
contains signatures verified by the most recent scan that covered
|
||||
them (a subsequent successful scan re-adds such files).
|
||||
- Database records outside the scanned operands are untouched, so
|
||||
disjoint trees can be scanned on different schedules into the same
|
||||
database.
|
||||
|
||||
`scan` runs **three sequential phases over the whole scan**. Parallelism lives
|
||||
inside each phase; batched database writes begin during the hash phase:
|
||||
`scan` runs **three sequential phases over the whole scan**.
|
||||
Parallelism lives inside each phase; batched database writes begin
|
||||
during the hash phase:
|
||||
|
||||
1. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
|
||||
with the walk worker pool: every operand seeds the shared queue, and each
|
||||
worker reads one directory at a time, handing discovered subdirectories back
|
||||
to the queue and running `lstat` on each regular file as it is discovered
|
||||
(while the directory's metadata is still hot). Sequential directory
|
||||
enumeration is metadata-latency-bound and takes hours at tens of millions of
|
||||
files; per-directory parallelism is what makes the walk tractable on large
|
||||
or busy pools. The walk builds the size census and resolves unchanged
|
||||
already-hashed files on the fly; every other file is carried to the hash
|
||||
phase as a (path, size, mtime) record.
|
||||
2. **hash** — with the census complete, each carried file's size decides its
|
||||
fate. Size-unique files are never read: new or changed ones are recorded
|
||||
without hashes in the update phase, unchanged unhashed ones simply keep
|
||||
their records. Every file with a shared size is hashed by the worker pool:
|
||||
read the first `min(1024, size)` bytes and the last `min(1024, size)` bytes
|
||||
(one read when `size <= 1024`, since the two windows coincide) and compute
|
||||
the SHA-256 of each. Zero-length files have constant hashes and are never
|
||||
opened. Files are hashed in **inode order** (minimizing seeks on spinning
|
||||
disks), and paths that are hard links to the same inode are **read once**,
|
||||
all sharing the one result — a hard-link backup farm costs one read per
|
||||
inode, not per path. The phase total counts actual reads, so progress and
|
||||
ETA are meaningful. Completed records are committed in batched transactions
|
||||
**while hashing runs**, so a scan interrupted after hours keeps everything
|
||||
hashed so far and the next scan resumes cheaply, skipping records already
|
||||
written.
|
||||
3. **update** — commit the final partial batch, the hash-less records for
|
||||
size-unique new and changed files, and the deletions for records the scan
|
||||
did not verify (vanished files, plus paths that failed to stat or hash).
|
||||
1. **walk + stat** — enumerate the trees under all `PATH` operands
|
||||
concurrently with the walk worker pool: every operand seeds the
|
||||
shared queue, and each worker reads one directory at a time,
|
||||
handing discovered subdirectories back to the queue and running
|
||||
`lstat` on each regular file as it is discovered (while the
|
||||
directory's metadata is still hot). Sequential directory
|
||||
enumeration is metadata-latency-bound and takes hours at tens of
|
||||
millions of files; per-directory parallelism is what makes the
|
||||
walk tractable on large or busy pools. The walk builds the size
|
||||
census and resolves unchanged already-hashed files on the fly;
|
||||
every other file is carried to the hash phase as a (path, size,
|
||||
mtime) record.
|
||||
2. **hash** — with the census complete, each carried file's size
|
||||
decides its fate. Size-unique files are never read: new or
|
||||
changed ones are recorded without hashes in the update phase,
|
||||
unchanged unhashed ones simply keep their records. Every file
|
||||
with a shared size is hashed by the worker pool: read the first
|
||||
`min(1024, size)` bytes and the last `min(1024, size)` bytes
|
||||
(one read when `size <= 1024`, since the two windows coincide)
|
||||
and compute the SHA-256 of each. Zero-length files have constant
|
||||
hashes and are never opened. Files are hashed in **inode order**
|
||||
(minimizing seeks on spinning disks), and paths that are hard
|
||||
links to the same inode are **read once**, all sharing the one
|
||||
result — a hard-link backup farm costs one read per inode, not
|
||||
per path. The phase total counts actual reads, so progress and
|
||||
ETA are meaningful. Completed records are committed in batched
|
||||
transactions **while hashing runs**, so a scan interrupted after
|
||||
hours keeps everything hashed so far and the next scan resumes
|
||||
cheaply, skipping records already written.
|
||||
3. **update** — commit the final partial batch, the hash-less
|
||||
records for size-unique new and changed files, and the deletions
|
||||
for records the scan did not verify (vanished files, plus paths
|
||||
that failed to stat or hash).
|
||||
|
||||
Rules for the walk:
|
||||
|
||||
- Only regular files. Skip directories, symlinks (do not follow, including
|
||||
symlink operands), sockets, FIFOs, and device nodes.
|
||||
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
|
||||
them would list every file once per snapshot).
|
||||
- Filesystem boundaries are crossed by default. With `-x` (long form
|
||||
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
|
||||
into a directory on a different filesystem than its `PATH` operand; each
|
||||
operand is bounded by its own filesystem.
|
||||
- On any per-path error (permission denied, file vanished between passes,
|
||||
unreadable): print a one-line warning to stderr, skip the path, and continue.
|
||||
Per-file errors never abort the run; the final summary reports how many were
|
||||
skipped. As specified above, a skipped path that has a database record from an
|
||||
earlier scan loses that record; an unreadable directory subtree likewise loses
|
||||
its records (accepted: the database mirrors what the latest scan could
|
||||
actually verify).
|
||||
- Only regular files. Skip directories, symlinks (do not follow,
|
||||
including symlink operands), sockets, FIFOs, and device nodes.
|
||||
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
|
||||
walking them would list every file once per snapshot).
|
||||
- Filesystem boundaries are crossed by default. With `-x`
|
||||
(long form `--one-file-system`, following the GNU `du`/`rsync`
|
||||
convention), never descend into a directory on a different
|
||||
filesystem than its `PATH` operand; each operand is bounded by its
|
||||
own filesystem.
|
||||
- On any per-path error (permission denied, file vanished between
|
||||
passes, unreadable): print a one-line warning to stderr, skip the
|
||||
path, and continue. Per-file errors never abort the run; the final
|
||||
summary reports how many were skipped. As specified above, a
|
||||
skipped path that has a database record from an earlier scan loses
|
||||
that record; an unreadable directory subtree likewise loses its
|
||||
records (accepted: the database mirrors what the latest scan could
|
||||
actually verify).
|
||||
|
||||
Concurrency: the walk phase (which also stats files) and the hash phase each use
|
||||
a worker pool of `--workers` workers (default `runtime.NumCPU()`); the walk
|
||||
parallelizes across directories, hashing across files. Both phases are
|
||||
seek-bound on spinning disks, so raising `--workers` well past the core count
|
||||
can help on pools with many spindles. The main goroutine owns partitioning,
|
||||
database writes, and progress rendering; progress display must never block the
|
||||
workers.
|
||||
Concurrency: the walk phase (which also stats files) and the hash
|
||||
phase each use a worker pool of `--workers` workers (default
|
||||
`runtime.NumCPU()`); the walk parallelizes across directories,
|
||||
hashing across files. Both phases are seek-bound on spinning disks,
|
||||
so raising `--workers` well past the core count can help on pools
|
||||
with many spindles. The main goroutine owns partitioning, database
|
||||
writes, and progress rendering; progress display must never block
|
||||
the workers.
|
||||
|
||||
`scan` writes nothing to stdout. The summary line on stderr reports the files
|
||||
seen this run broken down by disposition, plus skips:
|
||||
`scan` writes nothing to stdout. The summary line on stderr reports the
|
||||
files seen this run broken down by disposition, plus skips:
|
||||
|
||||
```
|
||||
scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
|
||||
```
|
||||
|
||||
(`removed` counts deleted database records, which are not part of the files-seen
|
||||
total.)
|
||||
(`removed` counts deleted database records, which are not part of the
|
||||
files-seen total.)
|
||||
|
||||
### `report` mode
|
||||
|
||||
`report` reads every record from the database and takes no positional arguments.
|
||||
`report` reads every record from the database and takes no positional
|
||||
arguments.
|
||||
|
||||
**`report` must never touch the filesystem being analyzed.** It does not stat,
|
||||
open, or otherwise access any path that appears in the records; its only I/O is
|
||||
reading the database and writing stdout/stderr. It must produce identical output
|
||||
whether or not the scanned filesystem is still mounted.
|
||||
**`report` must never touch the filesystem being analyzed.** It does not
|
||||
stat, open, or otherwise access any path that appears in the records; its
|
||||
only I/O is reading the database and writing stdout/stderr. It must
|
||||
produce identical output whether or not the scanned filesystem is still
|
||||
mounted.
|
||||
|
||||
Processing:
|
||||
|
||||
- Records without hashes (size-unique when last scanned) are excluded: their
|
||||
content is unknown, so they are never reported as duplicates.
|
||||
- Group the remaining records by the key `(size, head_hash, tail_hash)`.
|
||||
- Every group with two or more paths is a duplicate group.
|
||||
- Within each group, sort paths lexicographically (byte order). The first path
|
||||
is the group's `first`; every other path is a `dupe`.
|
||||
- Order groups by size descending (biggest reclaimable space first), tie-broken
|
||||
by `first` path ascending. Output must be fully deterministic for a given
|
||||
database state.
|
||||
- Records without hashes (size-unique when last scanned) are
|
||||
excluded: their content is unknown, so they are never reported as
|
||||
duplicates.
|
||||
- Group the remaining records by the key
|
||||
`(size, head_hash, tail_hash)`.
|
||||
- Every group with two or more paths is a duplicate group.
|
||||
- Within each group, sort paths lexicographically (byte order). The
|
||||
first path is the group's `first`; every other path is a `dupe`.
|
||||
- Order groups by size descending (biggest reclaimable space first),
|
||||
tie-broken by `first` path ascending. Output must be fully
|
||||
deterministic for a given database state.
|
||||
|
||||
#### Report output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
|
||||
group of N):
|
||||
TSV on stdout: a header line, then one row per duplicate file (N-1 rows
|
||||
for a group of N):
|
||||
|
||||
```
|
||||
first dupe size
|
||||
@@ -288,91 +322,99 @@ first dupe size
|
||||
/srv/a/big.iso /srv/c/big-copy2.iso 4294967296
|
||||
```
|
||||
|
||||
Summary to stderr: records read, number of duplicate groups, number of dupe
|
||||
files, and total reclaimable bytes (sum of `size` over all dupe rows) in human
|
||||
units.
|
||||
Summary to stderr: records read, number of duplicate groups, number of
|
||||
dupe files, and total reclaimable bytes (sum of `size` over all dupe
|
||||
rows) in human units.
|
||||
|
||||
### `trees` mode
|
||||
|
||||
`trees` reads the same database as `report` (no positional arguments) and
|
||||
reports **entire duplicate directory trees**: directories under which the exact
|
||||
same set of relative paths exists with the exact same file signatures.
|
||||
`trees` reads the same database as `report` (no positional arguments)
|
||||
and reports **entire duplicate directory trees**: directories under
|
||||
which the exact same set of relative paths exists with the exact same
|
||||
file signatures.
|
||||
|
||||
**`trees` must never touch the filesystem being analyzed** — the same rule as
|
||||
`report`. The directory hierarchy is reconstructed purely from the paths in the
|
||||
records, split on `/`.
|
||||
**`trees` must never touch the filesystem being analyzed** — the same
|
||||
rule as `report`. The directory hierarchy is reconstructed purely from
|
||||
the paths in the records, split on `/`.
|
||||
|
||||
Definitions:
|
||||
|
||||
- A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
|
||||
informational and excluded. An unhashed record (empty hashes) has unknown
|
||||
content: its signature is treated as unique to that file, so a tree containing
|
||||
an unhashed file never compares equal to any other tree.
|
||||
- A directory's **digest** is a SHA-256 Merkle digest computed bottom-up:
|
||||
serialize the directory's child entries — for a file child, its name and
|
||||
signature; for a subdirectory child, its name and that subdirectory's digest —
|
||||
sort the serialized entries byte-lexicographically, and hash the
|
||||
concatenation. Names are part of the digest: two trees whose files differ only
|
||||
in name are _not_ duplicates.
|
||||
- Two directories are **duplicate trees** when their digests are equal. Equal
|
||||
digests imply equal recursive file count and equal total byte size.
|
||||
- A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
|
||||
informational and excluded. An unhashed record (empty hashes) has
|
||||
unknown content: its signature is treated as unique to that file,
|
||||
so a tree containing an unhashed file never compares equal to any
|
||||
other tree.
|
||||
- A directory's **digest** is a SHA-256 Merkle digest computed
|
||||
bottom-up: serialize the directory's child entries — for a file
|
||||
child, its name and signature; for a subdirectory child, its name
|
||||
and that subdirectory's digest — sort the serialized entries
|
||||
byte-lexicographically, and hash the concatenation. Names are part
|
||||
of the digest: two trees whose files differ only in name are *not*
|
||||
duplicates.
|
||||
- Two directories are **duplicate trees** when their digests are
|
||||
equal. Equal digests imply equal recursive file count and equal
|
||||
total byte size.
|
||||
|
||||
Known limitation (accepted): hard-linked paths are reported as duplicates by
|
||||
`report` and count toward duplicate trees — their content is genuinely identical
|
||||
— even though they share storage, so removing one reclaims no space. Inode
|
||||
identity is used during the scan to avoid redundant reads but is not persisted
|
||||
in the database.
|
||||
Known limitation (accepted): hard-linked paths are reported as
|
||||
duplicates by `report` and count toward duplicate trees — their
|
||||
content is genuinely identical — even though they share storage, so
|
||||
removing one reclaims no space. Inode identity is used during the
|
||||
scan to avoid redundant reads but is not persisted in the database.
|
||||
|
||||
Known limitation (accepted): only regular files that appear in the database
|
||||
define a tree. Empty directories are invisible, and a file skipped during the
|
||||
scan (e.g. permission error) in one copy but not the other will make
|
||||
otherwise-identical trees compare as different.
|
||||
Known limitation (accepted): only regular files that appear in the
|
||||
database define a tree. Empty directories are invisible, and a file
|
||||
skipped during the scan (e.g. permission error) in one copy but not the
|
||||
other will make otherwise-identical trees compare as different.
|
||||
|
||||
Processing:
|
||||
|
||||
- Build the hierarchy, compute every directory's digest, and group directories
|
||||
by digest. Every group with two or more directories is a duplicate-tree group.
|
||||
- **Report only maximal trees.** A group is suppressed when its members' parents
|
||||
are pairwise distinct directories that all share a single digest — such a
|
||||
group is wholly implied by its parents' (or a further ancestor's) group.
|
||||
Groups containing sibling directories, or members whose parents differ, are
|
||||
always reported.
|
||||
- Within each group, sort paths lexicographically (byte order); the first path
|
||||
is `first`, every other path is a `dupe`.
|
||||
- Order groups by total tree size descending, tie-broken by `first` path
|
||||
ascending. Output must be fully deterministic for a given input.
|
||||
- Build the hierarchy, compute every directory's digest, and group
|
||||
directories by digest. Every group with two or more directories is a
|
||||
duplicate-tree group.
|
||||
- **Report only maximal trees.** A group is suppressed when its
|
||||
members' parents are pairwise distinct directories that all share a
|
||||
single digest — such a group is wholly implied by its parents' (or a
|
||||
further ancestor's) group. Groups containing sibling directories, or
|
||||
members whose parents differ, are always reported.
|
||||
- Within each group, sort paths lexicographically (byte order); the
|
||||
first path is `first`, every other path is a `dupe`.
|
||||
- Order groups by total tree size descending, tie-broken by `first`
|
||||
path ascending. Output must be fully deterministic for a given
|
||||
input.
|
||||
|
||||
#### Trees output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||||
group of N). `files` is the recursive regular-file count of one copy of the
|
||||
tree; `size` is the recursive total byte size of one copy:
|
||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows
|
||||
for a group of N). `files` is the recursive regular-file count of one
|
||||
copy of the tree; `size` is the recursive total byte size of one copy:
|
||||
|
||||
```
|
||||
first dupe files size
|
||||
/srv/a/project /srv/backup/project 3417 104857600
|
||||
```
|
||||
|
||||
Summary to stderr: records read, number of duplicate-tree groups, number of dupe
|
||||
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
|
||||
units.
|
||||
Summary to stderr: records read, number of duplicate-tree groups,
|
||||
number of dupe trees, and total reclaimable bytes (sum of `size` over
|
||||
all dupe rows) in human units.
|
||||
|
||||
### Progress
|
||||
|
||||
Use the progress-bar library for all scan progress; rendering in the style of
|
||||
`pv` is the model. All progress goes to stderr.
|
||||
Use the progress-bar library for all scan progress; rendering in the
|
||||
style of `pv` is the model. All progress goes to stderr.
|
||||
|
||||
Each phase gets its own display, rendered the moment the phase starts — a scan
|
||||
must never look hung. Loading the existing-record index (`load`) and the walk
|
||||
have no known totals while running: show a live count, rate, and elapsed time
|
||||
(spinner-style, no percentage or ETA). The hash and update phases have exact
|
||||
totals — only files that actually need hashing appear in the hash total, so its
|
||||
ETA is meaningful. Required elements for the bars with known totals:
|
||||
Each phase gets its own display, rendered the moment the phase
|
||||
starts — a scan must never look hung. Loading the existing-record
|
||||
index (`load`) and the walk have no known totals while running: show
|
||||
a live count, rate, and elapsed time (spinner-style, no percentage or
|
||||
ETA). The hash and update phases
|
||||
have exact totals — only files that actually need hashing appear in
|
||||
the hash total, so its ETA is meaningful. Required elements for the
|
||||
bars with known totals:
|
||||
|
||||
- elapsed time
|
||||
- estimated time remaining
|
||||
- a `[m/n] x%` display (items processed / total items, percent)
|
||||
- current rate (items/s)
|
||||
- elapsed time
|
||||
- estimated time remaining
|
||||
- a `[m/n] x%` display (items processed / total items, percent)
|
||||
- current rate (items/s)
|
||||
|
||||
Example shape (exact layout is flexible, content is not):
|
||||
|
||||
@@ -382,138 +424,147 @@ hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
|
||||
|
||||
Additional requirements:
|
||||
|
||||
- When stderr is not a TTY, do not emit ANSI redraws: print a plain one-line
|
||||
progress update no more often than every 5 seconds instead.
|
||||
- Progress updates are driven from the main goroutine and must be non-blocking
|
||||
with respect to the worker pool.
|
||||
- `report` and `trees` modes need no progress display, only their stderr
|
||||
summaries.
|
||||
- When stderr is not a TTY, do not emit ANSI redraws: print a plain
|
||||
one-line progress update no more often than every 5 seconds instead.
|
||||
- Progress updates are driven from the main goroutine and must be
|
||||
non-blocking with respect to the worker pool.
|
||||
- `report` and `trees` modes need no progress display, only their
|
||||
stderr summaries.
|
||||
|
||||
### Error handling and exit codes
|
||||
|
||||
- `0`: success, even if individual files were skipped with warnings.
|
||||
- `1`: fatal error (e.g., a `PATH` operand does not exist, the database cannot
|
||||
be created/opened/read/written, a missing database for `report`/`trees`,
|
||||
stdout write failure).
|
||||
- `2`: usage error (including `scan` with no `PATH` operand and `report`/`trees`
|
||||
with any positional argument).
|
||||
- `0`: success, even if individual files were skipped with warnings.
|
||||
- `1`: fatal error (e.g., a `PATH` operand does not exist, the
|
||||
database cannot be created/opened/read/written, a missing database
|
||||
for `report`/`trees`, stdout write failure).
|
||||
- `2`: usage error (including `scan` with no `PATH` operand and
|
||||
`report`/`trees` with any positional argument).
|
||||
|
||||
## Entrypoints
|
||||
|
||||
This repository adheres to the
|
||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||
standard: the normalized executables in `script/` are the entrypoints for the
|
||||
development workflow, and the `Makefile` targets are thin shims that call them.
|
||||
Every script is POSIX `sh`, resolves the repository root itself so it can be run
|
||||
from any working directory, and may be invoked directly. The provided
|
||||
entrypoints are:
|
||||
standard: the normalized executables in `script/` are the entrypoints
|
||||
for the development workflow, and the `Makefile` targets are thin
|
||||
shims that call them. Every script is POSIX `sh`, resolves the
|
||||
repository root itself so it can be run from any working directory,
|
||||
and may be invoked directly. The provided entrypoints are:
|
||||
|
||||
- `script/bootstrap` — install everything needed to build and develop this
|
||||
repository, idempotently, assuming nothing is present. `git`, `make`, `go`,
|
||||
and `node` come from the first of nix, apt, brew, or apk found on the host,
|
||||
and are presence-checked only; `node` is an unpinned host runtime like the
|
||||
rest, because nvm's prebuilt node is glibc-linked and does not run on this
|
||||
repo's musl/Alpine build image. The Markdown formatter itself — `prettier` —
|
||||
is pinned by `yarn.lock`'s integrity hash and installed with
|
||||
`yarn install --frozen-lockfile`. `golangci-lint` is deliberately **not**
|
||||
installed: it runs from a digest-pinned image via `script/lint` and never from
|
||||
a host install, so there is no host copy to drift from the pin. A missing
|
||||
`docker` is warned about rather than installed or treated as fatal —
|
||||
everything except linting works without it. Ends with `go mod download` and
|
||||
the `yarn` install.
|
||||
- `script/setup` — make a fresh clone ready for development: runs
|
||||
`script/bootstrap`, then `script/install-precommit`.
|
||||
- `script/projectname` — print this project's name (`sfdupes`). Scripts that
|
||||
need the name call it, so they stay identical across repositories.
|
||||
- `script/test` — run the test suite with a 30-second timeout and coverage
|
||||
enabled, rerunning verbosely on failure so the logs show which test failed.
|
||||
- `script/lint` — run the linter. It builds `Dockerfile.lint`, which copies the
|
||||
repository into the digest-pinned `golangci/golangci-lint` image and runs
|
||||
`golangci-lint config verify` and `golangci-lint run` as build steps, so a
|
||||
successful build is a clean lint. The linter is never run on the host, which
|
||||
makes a working `docker` the one prerequisite for linting — and therefore for
|
||||
`make check` and the pre-commit hook. Offline machines: the gate steps
|
||||
themselves make no network calls. `golangci-lint run` does not, and neither
|
||||
does `golangci-lint config verify` — it validates against a schema the pinned
|
||||
binary embeds, measured under `--network none` to both pass a valid config and
|
||||
reject an invalid one. The build around them does. `Dockerfile.lint` runs
|
||||
`go mod download` before the gates and this module has external dependencies,
|
||||
so a first lint on a machine with a cold BuildKit cache reaches the network
|
||||
there (as well as pulling the pinned image); under `--network none` it fails
|
||||
at that step, before any gate. That layer sits above the gates and stays
|
||||
cached, so once it is warm `script/lint` — and with it `make check` — runs
|
||||
entirely offline, until `go.mod` or `go.sum` changes and the download layer
|
||||
goes cold again. Because the daemon only ever sees a build context, this works
|
||||
when the docker daemon is remote and bind mounts are impossible.
|
||||
- `script/fmt` — format in place: `gofmt -s -w` for Go sources and `prettier`
|
||||
for Markdown (`--tab-width 4 --prose-wrap always`, the house settings, also
|
||||
carried in `.prettierrc`). prettier is the pinned devDependency in
|
||||
`package.json`/`yarn.lock`, installed by `script/bootstrap`.
|
||||
- `script/fmt-check` — the read-only counterpart of `script/fmt`: runs both
|
||||
checks, reports each independently so it is clear which failed, and exits
|
||||
non-zero if either found unformatted files instead of writing.
|
||||
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in
|
||||
that order. Modifies nothing. Needs `docker`, because `script/lint` does.
|
||||
- `script/docker` — build the Docker image, tagged with the name from
|
||||
`script/projectname`. The `Dockerfile` runs the gates as build steps, so this
|
||||
is also the check a developer or reviewer runs by hand.
|
||||
- `script/cibuild` — build the Docker image untagged. This is what the Gitea
|
||||
workflow runs on push; because the gates run as build steps, a successful
|
||||
build implies the repository is green.
|
||||
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a
|
||||
no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then
|
||||
`script/check`.
|
||||
- `script/install-precommit` — install the git pre-commit hook that runs
|
||||
`script/precommit`. The hook is written to the common git directory, so the
|
||||
main checkout and every worktree share it.
|
||||
- `script/verify-lint-image-pin` — fail unless the `golangci/golangci-lint`
|
||||
reference in `Dockerfile.lint` and the one in the `Dockerfile` lint stage are
|
||||
the same image at the same digest, naming both if not. The linter is pinned in
|
||||
those two files and nothing else keeps them in sync, so a bump applied to one
|
||||
alone would leave `make lint` and the `Dockerfile`'s fail-fast lint stage
|
||||
checking the same tree against different rulesets, both green. The guard
|
||||
restates neither pin — a third copy would be the same drift one file further
|
||||
out — and runs as a gate in both files, so `make lint`, `make check` and
|
||||
`make docker` all catch it.
|
||||
- `script/bootstrap` — install everything needed to build and
|
||||
develop this repository, idempotently, assuming nothing is
|
||||
present. `git`, `make`, and `go` come from the first of nix, apt,
|
||||
brew, or apk found on the host, and are presence-checked only.
|
||||
`golangci-lint` is deliberately **not** installed: it runs from a
|
||||
digest-pinned image via `script/lint` and never from a host
|
||||
install, so there is no host copy to drift from the pin. A missing
|
||||
`docker` is warned about rather than installed or treated as
|
||||
fatal — everything except linting works without it. Ends with
|
||||
`go mod download`.
|
||||
- `script/setup` — make a fresh clone ready for development: runs
|
||||
`script/bootstrap`, then `script/install-precommit`.
|
||||
- `script/projectname` — print this project's name (`sfdupes`).
|
||||
Scripts that need the name call it, so they stay identical across
|
||||
repositories.
|
||||
- `script/test` — run the test suite with a 30-second timeout and
|
||||
coverage enabled, rerunning verbosely on failure so the logs show
|
||||
which test failed.
|
||||
- `script/lint` — run the linter. It builds `Dockerfile.lint`, which
|
||||
copies the repository into the digest-pinned
|
||||
`golangci/golangci-lint` image and runs
|
||||
`golangci-lint config verify` and `golangci-lint run` as build
|
||||
steps, so a successful build is a clean lint. The linter is never
|
||||
run on the host, which makes a working `docker` the one
|
||||
prerequisite for linting — and therefore for `make check` and the
|
||||
pre-commit hook. Offline machines: the gate steps themselves make
|
||||
no network calls. `golangci-lint run` does not, and neither does
|
||||
`golangci-lint config verify` — it validates against a schema the
|
||||
pinned binary embeds, measured under `--network none` to both
|
||||
pass a valid config and reject an invalid one. The build around
|
||||
them does. `Dockerfile.lint` runs `go mod download` before the
|
||||
gates and this module has external dependencies, so a first lint
|
||||
on a machine with a cold BuildKit cache reaches the network there
|
||||
(as well as pulling the pinned image); under `--network none` it
|
||||
fails at that step, before any gate. That layer sits above the
|
||||
gates and stays cached, so once it is warm `script/lint` — and
|
||||
with it `make check` — runs entirely offline, until `go.mod` or
|
||||
`go.sum` changes and the download layer goes cold again. Because
|
||||
the daemon only ever sees a build context, this works when the
|
||||
docker daemon is remote and bind mounts are impossible.
|
||||
- `script/fmt` — format the Go sources in place (`gofmt -s -w`).
|
||||
Markdown is not formatted.
|
||||
- `script/fmt-check` — the read-only counterpart of `script/fmt`:
|
||||
prints any unformatted file and exits non-zero instead of writing.
|
||||
- `script/check` — run `script/test`, `script/lint`, and
|
||||
`script/fmt-check`, in that order. Modifies nothing. Needs
|
||||
`docker`, because `script/lint` does.
|
||||
- `script/docker` — build the Docker image, tagged with the name
|
||||
from `script/projectname`. The `Dockerfile` runs the gates as
|
||||
build steps, so this is also the check a developer or reviewer
|
||||
runs by hand.
|
||||
- `script/cibuild` — build the Docker image untagged. This is what
|
||||
the Gitea workflow runs on push; because the gates run as build
|
||||
steps, a successful build implies the repository is green.
|
||||
- `script/precommit` — run by the git pre-commit hook: `go mod tidy`
|
||||
must be a no-op (a resulting change to `go.mod` or `go.sum` fails
|
||||
the commit), then `script/check`.
|
||||
- `script/install-precommit` — install the git pre-commit hook that
|
||||
runs `script/precommit`. The hook is written to the common git
|
||||
directory, so the main checkout and every worktree share it.
|
||||
- `script/verify-lint-image-pin` — fail unless the
|
||||
`golangci/golangci-lint` reference in `Dockerfile.lint` and the
|
||||
one in the `Dockerfile` lint stage are the same image at the same
|
||||
digest, naming both if not. The linter is pinned in those two
|
||||
files and nothing else keeps them in sync, so a bump applied to
|
||||
one alone would leave `make lint` and the `Dockerfile`'s
|
||||
fail-fast lint stage checking the same tree against different
|
||||
rulesets, both green. The guard restates neither pin — a third
|
||||
copy would be the same drift one file further out — and runs as a
|
||||
gate in both files, so `make lint`, `make check` and `make docker`
|
||||
all catch it.
|
||||
|
||||
`script/verify-linter-pin` used to live here. It compared a linter binary
|
||||
against a version pin in `script/bootstrap`, and both of its subjects are gone:
|
||||
no linter binary is copied between build stages any more, and bootstrap pins no
|
||||
version because it installs no linter. The drift it existed to catch has moved
|
||||
from binary-versus-pin to pin-versus-pin, which is what
|
||||
`script/verify-lint-image-pin` above checks.
|
||||
`script/verify-linter-pin` used to live here. It compared a linter
|
||||
binary against a version pin in `script/bootstrap`, and both of its
|
||||
subjects are gone: no linter binary is copied between build stages any
|
||||
more, and bootstrap pins no version because it installs no linter. The
|
||||
drift it existed to catch has moved from binary-versus-pin to
|
||||
pin-versus-pin, which is what `script/verify-lint-image-pin` above
|
||||
checks.
|
||||
|
||||
`script/lint`, `script/docker` and `script/cibuild` all pass a freshly computed
|
||||
`CHECK_EPOCH` build argument, and the gate steps in `Dockerfile.lint` and
|
||||
`Dockerfile` reference it. Without that, an unchanged tree lets Docker serve the
|
||||
gate layers from cache and the build exits 0 having executed no tests and no
|
||||
lint — a green it never earned, and one this repository has produced twice.
|
||||
`CHECK_EPOCH` invalidates the gate layers on every run while leaving the pinned
|
||||
base images and the dependency layers cached. `script/lint`'s value carries the
|
||||
process id as well as the epoch, because two lint runs land inside the same
|
||||
second easily and a bare epoch would cache the second one.
|
||||
`script/lint`, `script/docker` and `script/cibuild` all pass a freshly
|
||||
computed `CHECK_EPOCH` build argument, and the gate steps in
|
||||
`Dockerfile.lint` and `Dockerfile` reference it. Without that, an
|
||||
unchanged tree lets Docker serve the gate layers from cache and the
|
||||
build exits 0 having executed no tests and no lint — a green it never
|
||||
earned, and one this repository has produced twice. `CHECK_EPOCH`
|
||||
invalidates the gate layers on every run while leaving the pinned base
|
||||
images and the dependency layers cached. `script/lint`'s value carries
|
||||
the process id as well as the epoch, because two lint runs land inside
|
||||
the same second easily and a bare epoch would cache the second one.
|
||||
|
||||
## Build
|
||||
|
||||
The `script/` entrypoints above are where the implementations live; the
|
||||
`Makefile` targets are shims onto them, except `build`, which carries the
|
||||
compile recipe:
|
||||
The `script/` entrypoints above are where the implementations live;
|
||||
the `Makefile` targets are shims onto them, except `build`, which
|
||||
carries the compile recipe:
|
||||
|
||||
- `make` / `make build` — build the `sfdupes` binary (cgo disabled); building is
|
||||
the default target.
|
||||
- `make bootstrap` — install the build and development dependencies.
|
||||
- `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook.
|
||||
- `make test` — run the test suite (30-second timeout; reruns with `-v` on
|
||||
failure).
|
||||
- `make lint` — run `golangci-lint` with the repo config, in Docker (see
|
||||
`script/lint`); requires `docker`.
|
||||
- `make fmt` / `make fmt-check` — format Go and Markdown sources / verify both
|
||||
without writing.
|
||||
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires
|
||||
`docker`, via `lint`.
|
||||
- `make docker` — build the Docker image, which runs the gates as build stages.
|
||||
- `make hooks` — install the pre-commit hook.
|
||||
- `make clean` — remove the binary.
|
||||
- `make` / `make build` — build the `sfdupes` binary (cgo
|
||||
disabled); building is the default target.
|
||||
- `make bootstrap` — install the build and development
|
||||
dependencies.
|
||||
- `make setup` — prepare a fresh clone: `bootstrap` plus the
|
||||
pre-commit hook.
|
||||
- `make test` — run the test suite (30-second timeout; reruns with
|
||||
`-v` on failure).
|
||||
- `make lint` — run `golangci-lint` with the repo config, in Docker
|
||||
(see `script/lint`); requires `docker`.
|
||||
- `make fmt` / `make fmt-check` — format Go sources / verify
|
||||
formatting without writing.
|
||||
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing.
|
||||
Requires `docker`, via `lint`.
|
||||
- `make docker` — build the Docker image, which runs the gates as
|
||||
build stages.
|
||||
- `make hooks` — install the pre-commit hook.
|
||||
- `make clean` — remove the binary.
|
||||
|
||||
### Definition of done
|
||||
|
||||
@@ -521,8 +572,8 @@ All of the following, run in this directory, must pass:
|
||||
|
||||
1. `make check` passes (tests, lint, `gofmt`).
|
||||
2. `make docker` succeeds.
|
||||
3. Smoke test — create a throwaway tree in a temp dir (never test against real
|
||||
data):
|
||||
3. Smoke test — create a throwaway tree in a temp dir (never test
|
||||
against real data):
|
||||
|
||||
```sh
|
||||
d=$(mktemp -d)
|
||||
@@ -554,29 +605,30 @@ All of the following, run in this directory, must pass:
|
||||
./sfdupes report
|
||||
```
|
||||
|
||||
(The scan database lives inside `$d` here purely for test hygiene; scanning
|
||||
`$d` therefore also records the SQLite file itself, which is harmless.)
|
||||
(The scan database lives inside `$d` here purely for test hygiene;
|
||||
scanning `$d` therefore also records the SQLite file itself, which
|
||||
is harmless.)
|
||||
|
||||
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
|
||||
group (two dupe rows, `first` is the lexicographically smallest path);
|
||||
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
|
||||
`t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
|
||||
`empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; groups
|
||||
ordered by size descending.
|
||||
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin`
|
||||
form one group (two dupe rows, `first` is the lexicographically
|
||||
smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group;
|
||||
`t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group;
|
||||
`tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and
|
||||
`tiny3` appear nowhere; groups ordered by size descending.
|
||||
|
||||
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
|
||||
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
|
||||
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
|
||||
differs by name).
|
||||
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
|
||||
`$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is
|
||||
suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
|
||||
appears nowhere (its file set differs by name).
|
||||
|
||||
Expected from the second `report` (after the modify/delete rescan):
|
||||
`one.bin` has left its group (its content changed), so
|
||||
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is gone from the
|
||||
database.
|
||||
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is
|
||||
gone from the database.
|
||||
|
||||
The test suite automates this scenario (see `scan_test.go`), plus a negative
|
||||
check: `report` and `trees` operate on the database alone and never touch
|
||||
the scanned filesystem.
|
||||
The test suite automates this scenario (see `scan_test.go`), plus a
|
||||
negative check: `report` and `trees` operate on the database alone
|
||||
and never touch the scanned filesystem.
|
||||
|
||||
## TODO
|
||||
|
||||
@@ -584,11 +636,12 @@ Tracked in [TODO.md](TODO.md).
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No full-content verification, no byte-for-byte compare, no deletion or linking
|
||||
of duplicates. The reports are advisory; acting on them is the user's job.
|
||||
- No persistence beyond the SQLite database described above; no export/import
|
||||
formats.
|
||||
- No daemon or filesystem watcher; scheduling rescans is cron's job.
|
||||
- No full-content verification, no byte-for-byte compare, no deletion
|
||||
or linking of duplicates. The reports are advisory; acting on them is
|
||||
the user's job.
|
||||
- No persistence beyond the SQLite database described above; no
|
||||
export/import formats.
|
||||
- No daemon or filesystem watcher; scheduling rescans is cron's job.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -1,386 +1,436 @@
|
||||
# Workflow
|
||||
|
||||
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
|
||||
tracker gets filed as an issue first
|
||||
- take an issue from the `1.0.0` milestone on the tracker; work not
|
||||
yet on the tracker gets filed as an issue first
|
||||
- branch (from `main`)
|
||||
- do the work, with tests, in small focused commits
|
||||
- record it at the top of Completed Steps (`TODO.md` changes in the same commit
|
||||
as the work)
|
||||
- push the branch and open a PR whose title ends with ` (closes #N)`
|
||||
- an independent review gates the merge; every finding is addressed or
|
||||
explicitly rebutted on the PR
|
||||
- record it at the top of Completed Steps (`TODO.md` changes in the
|
||||
same commit as the work)
|
||||
- push the branch and open a PR whose title ends with
|
||||
` (closes #N)`
|
||||
- an independent review gates the merge; every finding is addressed
|
||||
or explicitly rebutted on the PR
|
||||
- merge to `main` once the review passes
|
||||
|
||||
# Status
|
||||
|
||||
- pre-1.0
|
||||
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
|
||||
under the `1.0.0` milestone are what remains before the tag, and this file
|
||||
records history and process, not the queue
|
||||
- the Gitea tracker is authoritative for the pre-1.0 backlog: the
|
||||
open issues under the `1.0.0` milestone are what remains before
|
||||
the tag, and this file records history and process, not the queue
|
||||
|
||||
# Next Step
|
||||
|
||||
- take the next issue from the `1.0.0` milestone on the tracker:
|
||||
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
|
||||
of truth for what is left before 1.0.0. Individual issues are deliberately not
|
||||
restated here; a copy in this file drifts out of date the moment the tracker
|
||||
moves
|
||||
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
|
||||
the source of truth for what is left before 1.0.0. Individual
|
||||
issues are deliberately not restated here; a copy in this file
|
||||
drifts out of date the moment the tracker moves
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- restore Markdown formatting in `script/fmt`/`fmt-check` and reformat all
|
||||
Markdown to the house prettier settings (2026-09-21, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/19)
|
||||
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
|
||||
and `.dockerignore` (2026-09-21, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/22): `make clean` still
|
||||
removes the binary; the `.sqlite`/`-shm`/`-wal` ignores remain.
|
||||
|
||||
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
|
||||
`Dockerfile.lint` (2026-08-10, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
|
||||
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag,
|
||||
so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare
|
||||
`FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin`
|
||||
parses those `FROM` lines and still matches the tagless form; its advice line
|
||||
lost the now meaningless "tag and digest". With no tag in either reference, a
|
||||
tag-only disagreement no longer exists — a one-sided tag is caught as a plain
|
||||
mismatch.
|
||||
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the
|
||||
redundant tag, so both pins are the policy `# image:vX.Y.Z,
|
||||
YYYY-MM-DD` comment over a bare `FROM image@sha256:...`. Digest
|
||||
unchanged. `script/verify-lint-image-pin` parses those `FROM` lines
|
||||
and still matches the tagless form; its advice line lost the now
|
||||
meaningless "tag and digest". With no tag in either reference, a
|
||||
tag-only disagreement no longer exists — a one-sided tag is caught as
|
||||
a plain mismatch.
|
||||
|
||||
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
|
||||
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
|
||||
owner ruling, the linter runs inside a container invoked through the `script/`
|
||||
entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs
|
||||
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
|
||||
runs `golangci-lint config verify` and `golangci-lint run` as build steps, so
|
||||
a successful build IS a clean lint; `script/lint` is reduced to building it.
|
||||
`script/bootstrap` loses the `go install`, the pin constants, the version
|
||||
parser and `verify_golangci_lint` outright rather than hardening them — with
|
||||
nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that
|
||||
motivated them has no subject — and now warns rather than fails when `docker`
|
||||
is absent. Two traps handled. A lint build on an unchanged tree returns
|
||||
success in well under a second having run no linter, which is
|
||||
- run all linting in Docker via `Dockerfile.lint` and `script/lint`
|
||||
(2026-08-10, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling, the
|
||||
linter runs inside a container invoked through the `script/`
|
||||
entrypoint and is never installed on a host. New root
|
||||
`Dockerfile.lint` COPYs the repo into the digest-pinned
|
||||
`golangci/golangci-lint:v2.12.2` image and runs
|
||||
`golangci-lint config verify` and `golangci-lint run` as build
|
||||
steps, so a successful build IS a clean lint; `script/lint` is
|
||||
reduced to building it. `script/bootstrap` loses the `go install`,
|
||||
the pin constants, the version parser and `verify_golangci_lint`
|
||||
outright rather than hardening them — with nothing linting on the
|
||||
host, the `$GOPATH/bin` versus `PATH` problem that motivated them has
|
||||
no subject — and now warns rather than fails when `docker` is absent.
|
||||
Two traps handled. A lint build on an unchanged tree returns success
|
||||
in well under a second having run no linter, which is
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/32 and
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint`
|
||||
carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes
|
||||
the expanded command, not the declaration) and `script/lint` passes
|
||||
`"$(date +%s)-$$"` — the PID matters because two lint runs land inside the
|
||||
same second easily. And nothing inside an image build may shell out to docker,
|
||||
so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so
|
||||
`Dockerfile.lint` carries `ARG CHECK_EPOCH` referenced
|
||||
inside every gate `RUN` (BuildKit hashes the expanded command, not
|
||||
the declaration) and `script/lint` passes `"$(date +%s)-$$"` — the
|
||||
PID matters because two lint runs land inside the same second easily.
|
||||
And nothing inside an image build may shell out to docker, so the
|
||||
main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
|
||||
instead of `make lint`, and its build stage runs `make test` and
|
||||
`make fmt-check` instead of the `make check` aggregate (`make`, not the
|
||||
scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches
|
||||
what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by
|
||||
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge
|
||||
forcing BuildKit to finish linting before the build stage starts, and dropping
|
||||
it without replacing the edge would have ended fail-fast linting silently
|
||||
under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering
|
||||
edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the
|
||||
`go install` that justified it. `script/verify-linter-pin` is retired, deleted
|
||||
along with its README entry, because both of its subjects ceased to exist in
|
||||
the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION`
|
||||
in `script/bootstrap`, and there is now neither a binary crossing between
|
||||
stages nor a version pin in bootstrap. The drift it guarded has not gone away,
|
||||
it has moved — the linter is still pinned twice, now as the `FROM` line of
|
||||
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with
|
||||
nothing syncing them, which is exactly what
|
||||
`make fmt-check` instead of the `make check` aggregate (`make`, not
|
||||
the scripts bare, because the Makefile's `export CGO_ENABLED = 0`
|
||||
only reaches what it invokes). `COPY --from=lint`
|
||||
`/usr/bin/golangci-lint` is replaced by
|
||||
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the
|
||||
only edge forcing BuildKit to finish linting before the build stage
|
||||
starts, and dropping it without replacing the edge would have ended
|
||||
fail-fast linting silently under a still-green build. That is
|
||||
canonical `REPO_POLICIES.md:107`'s ordering edge, restored.
|
||||
`ENV PATH=/home/builder/go/bin:$PATH` is gone with the `go install`
|
||||
that justified it. `script/verify-linter-pin` is retired, deleted
|
||||
along with its README entry, because both of its subjects ceased to
|
||||
exist in the same change: it compared a linter binary against
|
||||
`GOLANGCI_LINT_VERSION` in `script/bootstrap`, and there is now
|
||||
neither a binary crossing between stages nor a version pin in
|
||||
bootstrap. The drift it guarded has not gone away, it has moved — the
|
||||
linter is still pinned twice, now as the `FROM` line of
|
||||
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage,
|
||||
with nothing syncing them, which is exactly what
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
|
||||
replacement is one new `script/verify-lint-image-pin`, run as a gate in both
|
||||
files, which compares the two references to each other and deliberately
|
||||
restates neither: a hardcoded expected digest would be a third copy and the
|
||||
same drift one file further out. `golangci-lint config verify` is included per
|
||||
the ruling, and the concern about its unpinned live HTTPS schema fetch was
|
||||
measured rather than assumed — under `--network none` the pinned binary both
|
||||
passes a valid config and rejects an invalid one with the jsonschema error, so
|
||||
it validates from an embedded schema and makes no network call of its own. The
|
||||
README scopes that to the gate steps rather than to linting as a whole:
|
||||
`Dockerfile.lint` runs `go mod download` above them, so a cold cache still
|
||||
needs the network and only a warm one lints offline. Verified: `make lint`
|
||||
green with every `PATH` directory containing a `golangci-lint` removed
|
||||
replacement is one new `script/verify-lint-image-pin`,
|
||||
run as a gate in both files, which compares the two references to
|
||||
each other and deliberately restates neither: a hardcoded expected
|
||||
digest would be a third copy and the same drift one file further out.
|
||||
`golangci-lint config verify` is included per the ruling, and the
|
||||
concern about its unpinned live HTTPS schema fetch was measured
|
||||
rather than assumed — under `--network none` the pinned binary both
|
||||
passes a valid config and rejects an invalid one with the jsonschema
|
||||
error, so it validates from an embedded schema and makes no network
|
||||
call of its own. The README scopes that to the gate steps rather
|
||||
than to linting as a whole: `Dockerfile.lint` runs `go mod download`
|
||||
above them, so a cold cache still needs the network and only a warm
|
||||
one lints offline. Verified: `make lint` green with every `PATH`
|
||||
directory containing a `golangci-lint` removed
|
||||
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
|
||||
`command -v golangci-lint` empty); two consecutive `script/lint` runs on an
|
||||
untouched tree both executed the linter, 27.7s and 28.7s in the lint step
|
||||
under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s
|
||||
and 41.8s wall clock — the no-cache rule was not weakened to shorten that.
|
||||
Negative control: a planted `var unusedIssue46Sentinel = 1` failed
|
||||
`script/lint` with
|
||||
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed
|
||||
`make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]`
|
||||
— `COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all
|
||||
zero occurrences — then reverted clean. The drift guard fails on a tag-only
|
||||
disagreement, on a digest-only disagreement, and on an unreadable reference,
|
||||
naming both sides. `make docker` green in 5m35s with all six gates executing
|
||||
under one epoch (lint 37.6s, test 25.2s reporting
|
||||
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The
|
||||
non-root quirk still holds: in the builder image with the Go test cache off,
|
||||
`--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the
|
||||
unprivileged user passes (exit 0). Noted for follow-up, not fixed here:
|
||||
`golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0
|
||||
in favour of `gomodguard_v2`.
|
||||
`command -v golangci-lint` empty); two consecutive `script/lint` runs
|
||||
on an untouched tree both executed the linter, 27.7s and 28.7s in the
|
||||
lint step under distinct epochs with the `COPY . .` layer `CACHED`
|
||||
above them, at 42.2s and 41.8s wall clock — the no-cache rule was not
|
||||
weakened to shorten that. Negative control: a planted
|
||||
`var unusedIssue46Sentinel = 1` failed `script/lint` with
|
||||
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and
|
||||
failed `make docker` at `[lint 9/9]` with the build stage stopped at
|
||||
`[builder 3/12]` — `COPY --from=lint`, `script/bootstrap`, the test
|
||||
gate and `make build` all zero occurrences — then reverted clean. The
|
||||
drift guard fails on a tag-only disagreement, on a digest-only
|
||||
disagreement, and on an unreadable reference, naming both sides.
|
||||
`make docker` green in 5m35s with all six gates executing under one
|
||||
epoch (lint 37.6s, test 25.2s reporting
|
||||
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`).
|
||||
The non-root quirk still holds: in the builder image with the Go test
|
||||
cache off, `--user 0:0` fails `TestScanHardlinkRunFailsTogether`
|
||||
(exit 1) where the unprivileged user passes (exit 0). Noted for
|
||||
follow-up, not fixed here: `golangci-lint` warns that the
|
||||
`gomodguard` linter is deprecated since v2.12.0 in favour of
|
||||
`gomodguard_v2`.
|
||||
|
||||
- install the Docker build stage's prerequisites by running `script/bootstrap`
|
||||
instead of `apk add --no-cache make` inline (2026-08-09, branch
|
||||
`dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires
|
||||
it, and the inline install left the build stage maintaining its own notion of
|
||||
the toolchain — exactly the divergence #24 exists to close, one layer down.
|
||||
The stage now copies `script/` plus `go.mod`/`go.sum` and runs
|
||||
`script/bootstrap`, which ends in `go mod download`, so the separate
|
||||
invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays,
|
||||
and moves above the bootstrap layer. It is the only edge making this stage
|
||||
depend on the lint stage, so deleting it as redundant would end fail-fast
|
||||
linting silently. Letting bootstrap install its own linter here would have
|
||||
reintroduced the second toolchain and paid for a from-source build of it. What
|
||||
makes the two stages provably one toolchain rather than two that happen to
|
||||
agree is a new `script/verify-linter-pin`, run in the build stage on the
|
||||
binary that arrives from the lint stage, before bootstrap: it fails the build
|
||||
naming both versions unless that binary is the version `script/bootstrap`
|
||||
pins. Bootstrap's own check could not serve that purpose — it reinstalls its
|
||||
pin from source and then verifies whatever `PATH` resolves, so drift
|
||||
self-heals silently and a lint stage image bumped on its own would lint at the
|
||||
new version while `make check` ran at the old one, green. The linter version
|
||||
is pinned in two independent places (the lint stage image digest and
|
||||
- install the Docker build stage's prerequisites by running
|
||||
`script/bootstrap` instead of `apk add --no-cache make` inline
|
||||
(2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical
|
||||
`REPO_POLICIES.md:97` requires it, and the inline install left the
|
||||
build stage maintaining its own notion of the toolchain — exactly
|
||||
the divergence #24 exists to close, one layer down. The stage now
|
||||
copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
|
||||
which ends in `go mod download`, so the separate invocation of that
|
||||
is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves
|
||||
above the bootstrap layer. It is the only edge making this stage
|
||||
depend on the lint stage, so deleting it as redundant would end
|
||||
fail-fast linting silently. Letting bootstrap install its own linter
|
||||
here would have reintroduced the second toolchain and paid for a
|
||||
from-source build of it. What makes the two stages provably one
|
||||
toolchain rather than two that happen to agree is a new
|
||||
`script/verify-linter-pin`, run in the build stage on the binary
|
||||
that arrives from the lint stage, before bootstrap: it fails the
|
||||
build naming both versions unless that binary is the version
|
||||
`script/bootstrap` pins. Bootstrap's own check could not serve that
|
||||
purpose — it reinstalls its pin from source and then verifies
|
||||
whatever `PATH` resolves, so drift self-heals silently and a lint
|
||||
stage image bumped on its own would lint at the new version while
|
||||
`make check` ran at the old one, green. The linter version is pinned
|
||||
in two independent places (the lint stage image digest and
|
||||
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
|
||||
half-applied bump is now a build failure. The pin is read out of
|
||||
`script/bootstrap`, which stays the single source of truth; a pin that cannot
|
||||
be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its
|
||||
only inputs are the copied binary and `script/`, so Docker invalidates the
|
||||
layer exactly when a cached result would stop being true, and it is documented
|
||||
with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because
|
||||
that is where bootstrap's `go install` lands and bootstrap verifies its
|
||||
installs against what `PATH` resolves — nothing in the image is shadowed by
|
||||
it, the directory does not exist until bootstrap runs. Everything added sits
|
||||
above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede
|
||||
`make check`. Verified: the guard fails the build with both versions named
|
||||
when the lint stage's linter is faked to a different version, and an
|
||||
unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and
|
||||
its `apk` branch, installing `git` and `make` and finding the copied linter
|
||||
already at the pin; a second build served the bootstrap and dependency layers
|
||||
`CACHED` while both gates ran with a fresh epoch; a planted `unused` finding
|
||||
failed the build at the lint gate in 48.9s with the build stage's `make check`
|
||||
never starting; and the suite run in the image as `--user 0:0` fails
|
||||
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is
|
||||
still load-bearing. That last check needs the Go test cache disabled — the
|
||||
first attempt reported `ok ... (cached)` as root, reusing the result the
|
||||
build-time run had left in the shared cache, which would have read as a pass.
|
||||
Build wall time, on a shared host running many concurrent builds and so noisy:
|
||||
2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source
|
||||
change, 5m14s cold. Only the cold one breaches the policy ceiling, and not
|
||||
because of this change — `chown -R builder:builder /src /home/builder` walks
|
||||
the module cache and re-runs on every source change, and it alone varied
|
||||
between 77s and 210s across those four builds, which is also the whole spread
|
||||
in the totals. The same cold measurement against `main` is 5m03s with a 209s
|
||||
`chown`. Filed as #43
|
||||
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
|
||||
`script/docker` cannot report a green they did not earn (2026-08-09, branch
|
||||
`cibuild-cache-bust`, closes #32): both scripts were bare `docker build`
|
||||
invocations with no cache control, and the `Dockerfile` copies the tree before
|
||||
running its gates, so on an unchanged tree Docker served those layers from
|
||||
cache and the build exited 0 having executed nothing. That is not hypothetical
|
||||
here — every merge this repo has done is a non-fast-forward merge of an
|
||||
undiverged branch, so each merge commit's tree is byte-identical to the branch
|
||||
head's and each merge CI run was almost certainly a full cache hit; and PR
|
||||
#31's reviewer found `make docker` returning success as a 17-layer cache hit,
|
||||
catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the
|
||||
scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or
|
||||
break it. `ARG` is scoped per stage and this `Dockerfile` has three gates
|
||||
across two — `make fmt-check` and `make lint` in the lint stage, `make check`
|
||||
in the build stage — so a single declaration would have left one stage
|
||||
silently cacheable; it is declared in both. And BuildKit hashes the expanded
|
||||
command, not the declaration, so a declared-but-unreferenced `ARG` invalidates
|
||||
nothing: each gate `RUN` echoes the epoch, which also puts the value in the
|
||||
build log as evidence the layer really ran. Placement is below the dependency
|
||||
layers on purpose — a build that goes cold every time would be a different
|
||||
bug, not a fix. Verified by running each script twice back to back on an
|
||||
unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on
|
||||
all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then
|
||||
61.1s; `script/docker` 61.1s then 53.4s), and twelve steps were still served
|
||||
`CACHED` in the steady state — both `go mod download`s, `apk add`, `adduser`,
|
||||
the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy out of
|
||||
the lint stage, and the binary copy into the runtime stage. The lint stage
|
||||
still gates the build stage: with a deliberate `unused` finding planted in the
|
||||
tree, the build failed at `make lint` in 36.1s and the build-stage
|
||||
`make check` never started. The build stage also still drops to the
|
||||
unprivileged `builder` user before `make check`, which the suite depends on
|
||||
rather than merely prefers: forcing the same image to run the tests as root
|
||||
fails `TestScanHardlinkRunFailsTogether`, because root reads straight through
|
||||
the `chmod(0)` the test uses to prove hard links are read once. This is the
|
||||
local fix only; propagating it to the canonical templates is `prompts` #26
|
||||
- check the installed golangci-lint version in `script/bootstrap` instead of
|
||||
only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24):
|
||||
`missing golangci-lint` meant any linter already on `PATH` satisfied the
|
||||
check, so the pin was never consulted and the v2.12.2 bump from #3 was inert
|
||||
on every host that already had one — this host ran v2.10.1 against a v2.12.2
|
||||
pin, `make check` went green, and `make docker` then rejected the same commit
|
||||
with findings the local gate never saw. The version now lives in one place,
|
||||
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a
|
||||
bump cannot half-apply; a `golangci_lint_version` helper parses
|
||||
`golangci-lint --version` (taking the field after the word `version` and
|
||||
tolerating an optional leading `v`, which the module ref carries and the
|
||||
binary's output does not), and any version that is not the pin — older, newer,
|
||||
absent or unparseable — is reinstalled. The install is then verified against
|
||||
the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
|
||||
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on
|
||||
`PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or
|
||||
the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow
|
||||
the install and leave the local gate disagreeing with CI under an affirmative
|
||||
`bootstrap complete`. Bootstrap now re-reads the effective version after
|
||||
installing and, on a mismatch, prints both paths and both versions to stderr
|
||||
and exits non-zero instead of claiming success; it does not reorder anyone's
|
||||
`script/bootstrap`, which stays the single source of truth; a pin
|
||||
that cannot be read is a hard failure, not a skip. The check needs
|
||||
no `CHECK_EPOCH`: its only inputs are the copied binary and
|
||||
`script/`, so Docker invalidates the layer exactly when a cached
|
||||
result would stop being true, and it is documented with the other
|
||||
entrypoints in the README. `$GOPATH/bin` joins `PATH` because
|
||||
that is where bootstrap's `go install` lands and bootstrap verifies
|
||||
its installs against what `PATH` resolves — nothing in the image is
|
||||
shadowed by it, the directory does not exist until bootstrap runs.
|
||||
Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and
|
||||
`USER builder` still precede `make check`. Verified: the guard fails
|
||||
the build with both versions named when the lint stage's linter is
|
||||
faked to a different version, and an unmodified build still passes
|
||||
it; bootstrap runs clean under Alpine's `sh` and its `apk` branch,
|
||||
installing `git` and `make` and finding the copied
|
||||
linter already at the pin; a second build served the bootstrap and
|
||||
dependency layers `CACHED` while both gates ran with a fresh epoch;
|
||||
a planted `unused` finding failed the build at the lint gate in
|
||||
48.9s with the build stage's `make check` never starting; and the
|
||||
suite run in the image as `--user 0:0` fails
|
||||
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged
|
||||
user is still load-bearing. That last check needs the Go test cache
|
||||
disabled — the first attempt reported `ok ... (cached)` as root,
|
||||
reusing the result the build-time run had left in the shared cache,
|
||||
which would have read as a pass. Build wall time, on a shared host
|
||||
running many concurrent builds and so noisy: 2m13s on an unchanged
|
||||
tree, 2m17s and 4m29s for two builds after a source change, 5m14s
|
||||
cold. Only the cold one breaches the policy ceiling, and not because
|
||||
of this change — `chown -R builder:builder /src /home/builder` walks
|
||||
the module cache and re-runs on every source change, and it alone
|
||||
varied between 77s and 210s across those four builds, which is also
|
||||
the whole spread in the totals. The same cold measurement against
|
||||
`main` is 5m03s with a 209s `chown`. Filed as #43
|
||||
- bust the Docker layer cache for the gate steps, so `script/cibuild`
|
||||
and `script/docker` cannot report a green they did not earn
|
||||
(2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts
|
||||
were bare `docker build` invocations with no cache control, and the
|
||||
`Dockerfile` copies the tree before running its gates, so on an
|
||||
unchanged tree Docker served those layers from cache and the build
|
||||
exited 0 having executed nothing. That is not hypothetical here —
|
||||
every merge this repo has done is a non-fast-forward merge of an
|
||||
undiverged branch, so each merge commit's tree is byte-identical to
|
||||
the branch head's and each merge CI run was almost certainly a full
|
||||
cache hit; and PR #31's reviewer found `make docker` returning
|
||||
success as a 17-layer cache hit, catching it only by being
|
||||
suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing
|
||||
`--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break
|
||||
it. `ARG` is scoped per stage and this `Dockerfile` has three gates
|
||||
across two — `make fmt-check` and `make lint` in the lint stage,
|
||||
`make check` in the build stage — so a single declaration would have
|
||||
left one stage silently cacheable; it is declared in both. And
|
||||
BuildKit hashes the expanded command, not the declaration, so a
|
||||
declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN`
|
||||
echoes the epoch, which also puts the value in the build log as
|
||||
evidence the layer really ran. Placement is below the dependency
|
||||
layers on purpose — a build that goes cold every time would be a
|
||||
different bug, not a fix. Verified by running each script twice back
|
||||
to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all
|
||||
three gates executed on all four runs, each with a fresh epoch in
|
||||
the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s
|
||||
then 53.4s), and twelve steps were still served `CACHED` in the
|
||||
steady state — both `go mod download`s, `apk add`, `adduser`, the
|
||||
`chown`, every `go.mod`/`go.sum` and source copy, the linter copy
|
||||
out of the lint stage, and the binary copy into the runtime stage.
|
||||
The lint stage still gates the build stage: with a deliberate
|
||||
`unused` finding planted in the tree, the build failed at
|
||||
`make lint` in 36.1s and the build-stage `make check` never started.
|
||||
The build stage also still drops to the unprivileged `builder` user
|
||||
before `make check`, which the suite depends on rather than merely
|
||||
prefers: forcing the same image to run the tests as root fails
|
||||
`TestScanHardlinkRunFailsTogether`, because root reads straight
|
||||
through the `chmod(0)` the test uses to prove hard links are read
|
||||
once. This is the local fix only; propagating it to the canonical
|
||||
templates is `prompts` #26
|
||||
- check the installed golangci-lint version in `script/bootstrap`
|
||||
instead of only its presence (2026-08-09, branch
|
||||
`bootstrap-version-check`, closes #24): `missing golangci-lint` meant
|
||||
any linter already on `PATH` satisfied the check, so the pin was never
|
||||
consulted and the v2.12.2 bump from #3 was inert on every host that
|
||||
already had one — this host ran v2.10.1 against a v2.12.2 pin,
|
||||
`make check` went green, and `make docker` then rejected the same
|
||||
commit with findings the local gate never saw. The version now lives
|
||||
in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module
|
||||
ref derived from it so a bump cannot half-apply; a
|
||||
`golangci_lint_version` helper parses `golangci-lint --version`
|
||||
(taking the field after the word `version` and tolerating an optional
|
||||
leading `v`, which the module ref carries and the binary's output does
|
||||
not), and any version that is not the pin — older, newer, absent or
|
||||
unparseable — is reinstalled. The install is then verified against the
|
||||
binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
|
||||
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
|
||||
first on `PATH`, so a wrong-version one sitting ahead of it — nix,
|
||||
apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
|
||||
stage makes — would swallow the install and leave the local gate
|
||||
disagreeing with CI under an affirmative `bootstrap complete`.
|
||||
Bootstrap now re-reads the effective version after installing and, on
|
||||
a mismatch, prints both paths and both versions to stderr and exits
|
||||
non-zero instead of claiming success; it does not reorder anyone's
|
||||
`PATH` or delete their binary. The `--version` call keeps its stderr
|
||||
connected, so a present-but-broken binary says why rather than reinstalling
|
||||
forever in silence, and is bounded by `timeout(1)` where that exists, so a
|
||||
wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their
|
||||
presence-only checks and now say why in a comment: they are host
|
||||
package-manager tools the repo deliberately does not pin, with `go.mod`
|
||||
governing the language version and the digest-pinned images covering
|
||||
reproducible builds. Verified on this host by bootstrapping from v2.10.1 to
|
||||
v2.12.2 and running it again to a no-op, plus stub runs of the real script
|
||||
under `dash` covering a thirteen-input parse matrix (absent, older, newer,
|
||||
host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit,
|
||||
impostor binary, `(devel)`, trailing `version`), a shadowed install that must
|
||||
exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a
|
||||
wedged binary that must hit the timeout; `make check` and `make lint` are
|
||||
clean at v2.12.2, so v2.10.1 was not hiding any findings on `main`
|
||||
connected, so a present-but-broken binary says why rather than
|
||||
reinstalling forever in silence, and is bounded by `timeout(1)` where
|
||||
that exists, so a wedged binary cannot hang bootstrap. `git`, `make`
|
||||
and `go` keep their presence-only checks and now say why in a
|
||||
comment: they are host package-manager tools the repo deliberately
|
||||
does not pin, with `go.mod` governing the language version and the
|
||||
digest-pinned images covering reproducible builds. Verified on this
|
||||
host by bootstrapping from v2.10.1 to v2.12.2 and running it again to
|
||||
a no-op, plus stub runs of the real script under `dash` covering a
|
||||
thirteen-input parse matrix (absent, older, newer, host-style,
|
||||
image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor
|
||||
binary, `(devel)`, trailing `version`), a shadowed install that must
|
||||
exit non-zero, an install destination not on `PATH` at all, `GOBIN`
|
||||
set, and a wedged binary that must hit the timeout; `make check` and
|
||||
`make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
|
||||
findings on `main`
|
||||
- unwind the hash worker pool on the error path (2026-08-09, branch
|
||||
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment
|
||||
`recordRun` failed and abandon the pool — the feeder parked forever on a full
|
||||
`jobs` channel and every worker on a full `results` channel. That only stopped
|
||||
being invisible when #4 landed and `runScan` began unwinding instead of
|
||||
calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every
|
||||
blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is
|
||||
closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels
|
||||
and then drains `results` until the last goroutine has exited — draining is
|
||||
what frees a worker already parked on a send. `ctx` is threaded from
|
||||
`cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole
|
||||
database layer (it is the first parameter everywhere), so #5 can hand this
|
||||
path a signal and needs to add nothing else. The walk pool never leaked,
|
||||
because `walkPhase` always drains its events to close, but it has the same
|
||||
unbounded-send shape and #5 will give it an early return, so it gets the same
|
||||
treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a
|
||||
partial size census, and every file it never reached looks vanished to the
|
||||
update phase. That phase's own `BeginTx` fails on the same cancelled context
|
||||
before deleting anything, so the guard is defence in depth rather than the
|
||||
only barrier — but it is the one that survives #5 deciding an interrupted scan
|
||||
may commit what it has. Tests drive `run(scan)` against a database whose
|
||||
insert trigger aborts, and assert both that the scan fails instead of hanging
|
||||
and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a
|
||||
second set cancels a scan part-way through the walk — deterministically, by
|
||||
counting the scan's own consultations of `ctx.Done()` rather than racing a
|
||||
timer — and asserts that it stops at the guard holding a partial census and a
|
||||
still-populated record index, with every record intact. The remaining
|
||||
cancellation branches of both pools are covered by direct tests of
|
||||
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
|
||||
and `hashPhase`
|
||||
- guarantee the database is closed on every fatal exit path (2026-08-09, branch
|
||||
`db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so
|
||||
the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now
|
||||
actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`,
|
||||
`loadRecords` and `resolveRoots` return errors instead. The single exit point
|
||||
is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned)
|
||||
to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a
|
||||
runtime failure from being reported as a usage error or printing the usage
|
||||
text. New `main_test.go` drives the CLI in-process and asserts the exit codes
|
||||
from README §Error handling plus the stdout/stderr split, including that a
|
||||
fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar
|
||||
behind for `scan`, `report` or `trees`
|
||||
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
|
||||
`golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter
|
||||
in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2,
|
||||
and replaced `.golangci.yml` with the canonical file — the linter settings
|
||||
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the
|
||||
moment `recordRun` failed and abandon the pool — the feeder parked
|
||||
forever on a full `jobs` channel and every worker on a full
|
||||
`results` channel. That only stopped being invisible when #4 landed
|
||||
and `runScan` began unwinding instead of calling `os.Exit`. The
|
||||
pool is now an owned, context-aware `hashPool`: every blocking send
|
||||
in the feeder and the workers selects on `ctx.Done()`, `jobs` is
|
||||
closed on every path out, and `hashPhase` defers `pool.stop()`,
|
||||
which cancels and then drains `results` until the last goroutine
|
||||
has exited — draining is what frees a worker already parked on a
|
||||
send. `ctx` is threaded from `cmd.Context()` through `runScan`,
|
||||
`syncScan`, both worker pools and the whole database layer (it is
|
||||
the first parameter everywhere), so #5 can hand this path a signal
|
||||
and needs to add nothing else. The walk pool never leaked, because
|
||||
`walkPhase` always drains its events to close, but it has the same
|
||||
unbounded-send shape and #5 will give it an early return, so it
|
||||
gets the same treatment plus a `ctx.Err()` guard after the walk: a
|
||||
cancelled walk yields a partial size census, and every file it never
|
||||
reached looks vanished to the update phase. That phase's own
|
||||
`BeginTx` fails on the same cancelled context before deleting
|
||||
anything, so the guard is defence in depth rather than the only
|
||||
barrier — but it is the one that survives #5 deciding an interrupted
|
||||
scan may commit what it has. Tests drive `run(scan)` against a
|
||||
database whose insert trigger aborts, and assert both that the scan
|
||||
fails instead of hanging and that `runtime.NumGoroutine()` polls
|
||||
back to its pre-scan baseline; a second set cancels a scan part-way
|
||||
through the walk — deterministically, by counting the scan's own
|
||||
consultations of `ctx.Done()` rather than racing a timer — and
|
||||
asserts that it stops at the guard holding a partial census and a
|
||||
still-populated record index, with every record intact. The
|
||||
remaining cancellation branches of both pools are covered by direct
|
||||
tests of `sendEvent`, the walk workers, `dispatchDirs`,
|
||||
`feedHashJobs`, `hashWorker` and `hashPhase`
|
||||
- guarantee the database is closed on every fatal exit path
|
||||
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
|
||||
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
|
||||
it the SQLite WAL checkpoint — now actually runs when a subcommand
|
||||
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
|
||||
`resolveRoots` return errors instead. The single exit point is `run`
|
||||
in `main.go`: it maps a `fatalError` (anything a subcommand
|
||||
returned) to exit 1 and cobra's own argument and flag errors to exit
|
||||
2, which keeps a runtime failure from being reported as a usage
|
||||
error or printing the usage text. New `main_test.go` drives the CLI
|
||||
in-process and asserts the exit codes from README §Error handling
|
||||
plus the stdout/stderr split, including that a fatal error raised
|
||||
after the database is open leaves no `-wal`/`-shm` sidecar behind
|
||||
for `scan`, `report` or `trees`
|
||||
- update golangci-lint to v2.12.2 with the canonical config
|
||||
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
|
||||
closes #3): bumped the pinned linter in the `Dockerfile` lint
|
||||
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
|
||||
`.golangci.yml` with the canonical file — the linter settings
|
||||
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
|
||||
`linters.settings` per the v2 schema, so they are actually applied; no new
|
||||
lint findings surfaced
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
|
||||
like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12
|
||||
`script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`,
|
||||
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
|
||||
`install-precommit`) and every Makefile target is now a thin shim over them,
|
||||
matching the other managed repos
|
||||
`linters.settings` per the v2 schema, so they are actually
|
||||
applied; no new lint findings surfaced
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
||||
entrypoints like the other managed repos (2026-07-26, commit
|
||||
`3abeacf`, closes #1): all 12 `script/` entrypoints exist
|
||||
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
|
||||
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
|
||||
`install-precommit`) and every Makefile target is now a thin shim
|
||||
over them, matching the other managed repos
|
||||
- make the binary the default Make target (2026-07-24, branch
|
||||
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran
|
||||
`check` plus `build`); `make build` remains as an alias
|
||||
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
|
||||
`scan-wide-phases`): all operands seed the shared walk pool and every pass
|
||||
runs once over the whole scan, so totals and ETAs are scan-global; the
|
||||
per-operand walk/hash/update cycles and their stderr announcements are gone;
|
||||
the update pass commits in batched transactions — the filesystem is
|
||||
authoritative and the database an eventually-consistent reflection, so
|
||||
scan-level atomicity is not required
|
||||
`make-default-target`): plain `make` now builds `sfdupes`
|
||||
(previously it ran `check` plus `build`); `make build` remains as
|
||||
an alias
|
||||
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
|
||||
branch `scan-wide-phases`): all operands seed the shared walk pool
|
||||
and every pass runs once over the whole scan, so totals and ETAs
|
||||
are scan-global; the per-operand walk/hash/update cycles and their
|
||||
stderr announcements are gone; the update pass commits in batched
|
||||
transactions — the filesystem is authoritative and the database an
|
||||
eventually-consistent reflection, so scan-level atomicity is not
|
||||
required
|
||||
- split the stat pass back out of the walk (2026-07-24, branch
|
||||
`parallel-phases`): phases are strictly sequential again — walk, stat, hash,
|
||||
update per operand — with parallelism only inside each phase; the walk
|
||||
enumerates paths with per-directory workers and the stat pass lstats them with
|
||||
per-file workers, restoring the exact total/ETA stat bar
|
||||
- announce each operand on stderr before its passes (2026-07-24, branch
|
||||
`scan-operand-progress`): with per-operand walk/hash/update cycles, a
|
||||
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
|
||||
whole run's — an operator watching operand 3 of 14 hash 300k files concluded
|
||||
20M files were being skipped
|
||||
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
|
||||
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
|
||||
4h on a ZFS server); it is now a per-directory worker-pool traversal that
|
||||
records size/mtime during the walk (folding away the separate stat pass,
|
||||
halving metadata I/O), and each `PATH` operand commits in its own transaction
|
||||
so an interrupted scan keeps completed operands
|
||||
`parallel-phases`): phases are strictly sequential again — walk,
|
||||
stat, hash, update per operand — with parallelism only inside each
|
||||
phase; the walk enumerates paths with per-directory workers and the
|
||||
stat pass lstats them with per-file workers, restoring the exact
|
||||
total/ETA stat bar
|
||||
- announce each operand on stderr before its passes (2026-07-24,
|
||||
branch `scan-operand-progress`): with per-operand walk/hash/update
|
||||
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
|
||||
that looked like the whole run's — an operator watching operand 3 of
|
||||
14 hash 300k files concluded 20M files were being skipped
|
||||
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
|
||||
was a single goroutine and took hours at ~20M files on a busy pool
|
||||
(observed: 22M files in 4h on a ZFS server); it is now a
|
||||
per-directory worker-pool traversal that records size/mtime during
|
||||
the walk (folding away the separate stat pass, halving metadata
|
||||
I/O), and each `PATH` operand commits in its own transaction so an
|
||||
interrupted scan keeps completed operands
|
||||
|
||||
- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
|
||||
now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
|
||||
disabled) keyed by absolute path that survives between runs — a rescan hashes
|
||||
only new or changed files (by mtime/size), deletes records for files vanished
|
||||
from under the scanned operands, and leaves records outside them untouched, so
|
||||
`scan` can be cronned daily; `report` and `trees` read the database (no
|
||||
positional arguments) instead of a scan stream. Database at
|
||||
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
|
||||
journaling plus a single-transaction update keep a report run during a scan
|
||||
safe
|
||||
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
|
||||
and push `main` plus tags (2026-07-23)
|
||||
- persistent scan database (2026-07-24, branch `persistent-database`):
|
||||
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
|
||||
Go, cgo stays disabled) keyed by absolute path that survives between
|
||||
runs — a rescan hashes only new or changed files (by mtime/size),
|
||||
deletes records for files vanished from under the scanned operands,
|
||||
and leaves records outside them untouched, so `scan` can be cronned
|
||||
daily; `report` and `trees` read the database (no positional
|
||||
arguments) instead of a scan stream. Database at
|
||||
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
|
||||
WAL journaling plus a single-transaction update keep a report run
|
||||
during a scan safe
|
||||
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
|
||||
`v0.0.1`, and push `main` plus tags (2026-07-23)
|
||||
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
||||
`PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
|
||||
`-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
|
||||
boundaries, which are crossed by default
|
||||
`PATH...` operands via cobra flags replacing the `/srv` `-root`
|
||||
default; new `-x`/`--one-file-system` flag (GNU convention) to stop
|
||||
at filesystem boundaries, which are crossed by default
|
||||
- bring the repo into full policy compliance (2026-07-23, branch
|
||||
`repo-policy-compliance`; checklist below)
|
||||
- `git init` with README-only first commit; code baseline committed on `main`
|
||||
(2026-07-22)
|
||||
- `git init` with README-only first commit; code baseline committed on
|
||||
`main` (2026-07-22)
|
||||
- implement `scan`, `report`, and `trees` subcommands (pre-git history)
|
||||
|
||||
# Future Steps
|
||||
|
||||
- possible later features (explicitly out of scope per README): full-content
|
||||
verification of candidates, removal-script helpers
|
||||
- possible later features (explicitly out of scope per README):
|
||||
full-content verification of candidates, removal-script helpers
|
||||
|
||||
# Repo Policy Compliance
|
||||
|
||||
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
|
||||
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
|
||||
formatting commit is needed.
|
||||
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
|
||||
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
|
||||
standalone formatting commit is needed.
|
||||
|
||||
- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
|
||||
untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
|
||||
- [x] `.gitignore` missing — the compiled `sfdupes` binary and
|
||||
`files.dat` sit untracked in the tree; needs OS/editor/Go
|
||||
artifacts plus secrets patterns
|
||||
- [x] `.editorconfig` missing
|
||||
- [x] `LICENSE` missing and README has no License section (MIT assumed from
|
||||
house convention — user to confirm)
|
||||
- [x] `LICENSE` missing and README has no License section (MIT assumed
|
||||
from house convention — user to confirm)
|
||||
- [x] `REPO_POLICIES.md` missing from repo root
|
||||
- [x] `.golangci.yml` missing (install canonical copy); code must then pass
|
||||
`make lint` (150 findings fixed; `make lint` is clean)
|
||||
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
|
||||
`docker`, `hooks`; `check` currently depends on `build`, which writes the
|
||||
binary (`make check` must not modify files)
|
||||
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
|
||||
with a 30-second timeout and the conditional `-v` rerun pattern (suite
|
||||
covers parsing, grouping, digests, suppression, hashing, and the scan
|
||||
pipeline; 64% coverage)
|
||||
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
|
||||
lint stage, build stage running `make check`
|
||||
- [x] `.golangci.yml` missing (install canonical copy); code must then
|
||||
pass `make lint` (150 findings fixed; `make lint` is clean)
|
||||
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
|
||||
`fmt-check`, `docker`, `hooks`; `check` currently depends on
|
||||
`build`, which writes the binary (`make check` must not modify
|
||||
files)
|
||||
- [x] no tests — `go test ./...` has nothing to run; policy requires
|
||||
real tests with a 30-second timeout and the conditional `-v`
|
||||
rerun pattern (suite covers parsing, grouping, digests,
|
||||
suppression, hashing, and the scan pipeline; 64% coverage)
|
||||
- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
|
||||
fail-fast lint stage, build stage running `make check`
|
||||
- [x] `.dockerignore` missing
|
||||
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
|
||||
action pinned by commit SHA)
|
||||
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
|
||||
checkout action pinned by commit SHA)
|
||||
- [x] README lacks required sections: Description first line
|
||||
(name/purpose/category/license/author), Getting Started, Rationale, TODO,
|
||||
License, Author
|
||||
- [x] README non-goal "no git repository setup and no CI" is stale now that the
|
||||
repo is under git with CI
|
||||
- [x] pre-commit hook not installed (`make hooks` once the target exists)
|
||||
(name/purpose/category/license/author), Getting Started,
|
||||
Rationale, TODO, License, Author
|
||||
- [x] README non-goal "no git repository setup and no CI" is stale now
|
||||
that the repo is under git with CI
|
||||
- [x] pre-commit hook not installed (`make hooks` once the target
|
||||
exists)
|
||||
|
||||
Accepted divergences (no action):
|
||||
|
||||
- flat single-package layout with `.go` files in the repo root — fine for a
|
||||
small single-binary tool per the Go styleguide; the tracker audit agrees
|
||||
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go
|
||||
builds) and the race detector requires cgo
|
||||
- flat single-package layout with `.go` files in the repo root — fine
|
||||
for a small single-binary tool per the Go styleguide; the tracker
|
||||
audit agrees
|
||||
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
|
||||
(pure-Go builds) and the race detector requires cgo
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
{
|
||||
"devDependencies": {
|
||||
"prettier": "3.8.1"
|
||||
}
|
||||
}
|
||||
@@ -11,12 +11,6 @@ set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
# yarn provides prettier, which formats Markdown. yarn is a tool, like
|
||||
# node/git/make/go below; the reference that governs formatting output is
|
||||
# prettier, pinned by yarn.lock's integrity hash and installed by
|
||||
# `yarn install --frozen-lockfile`.
|
||||
YARN_VERSION="1.22.22"
|
||||
|
||||
PKGMGR=""
|
||||
SUDO=""
|
||||
APT_UPDATED=""
|
||||
@@ -64,21 +58,6 @@ missing() {
|
||||
! command -v "$1" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
ensure_node() {
|
||||
if ! missing node; then return 0; fi
|
||||
pkg_install nodejs nodejs node nodejs
|
||||
}
|
||||
|
||||
ensure_yarn() {
|
||||
if ! missing yarn; then return 0; fi
|
||||
if ! missing corepack; then
|
||||
corepack enable >/dev/null 2>&1 || true
|
||||
corepack prepare "yarn@$YARN_VERSION" --activate
|
||||
else
|
||||
pkg_install yarn yarn yarn yarn
|
||||
fi
|
||||
}
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
|
||||
@@ -92,16 +71,6 @@ main() {
|
||||
if missing make; then pkg_install gnumake make make make; fi
|
||||
if missing go; then pkg_install go golang go go; fi
|
||||
|
||||
# node runs prettier and is an unpinned host tool for the same reason
|
||||
# git/make/go are: it comes from the host package manager, whatever
|
||||
# version it ships. It is not installed via nvm the way the canonical
|
||||
# template does, because nvm's prebuilt node is glibc-linked and does
|
||||
# not run on this repo's musl/Alpine build image. prettier — the tool
|
||||
# whose version affects formatting output — is pinned by yarn.lock.
|
||||
ensure_node
|
||||
ensure_yarn
|
||||
yarn install --frozen-lockfile
|
||||
|
||||
# Linting runs via docker only (script/lint), so docker is a lint
|
||||
# prerequisite rather than something bootstrap installs. Warn, do
|
||||
# not fail: everything except `make lint` — and, through it,
|
||||
|
||||
+1
-12
@@ -1,23 +1,12 @@
|
||||
#!/bin/sh
|
||||
# script/fmt: format all files (writes). gofmt for Go, prettier for
|
||||
# Markdown. prettier is the pinned devDependency in package.json/
|
||||
# yarn.lock; script/bootstrap installs it (see run_prettier).
|
||||
# script/fmt: format all files (writes).
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
run_prettier() {
|
||||
if ! command -v yarn >/dev/null 2>&1; then
|
||||
echo "fmt: yarn not found; run script/bootstrap first" >&2
|
||||
exit 1
|
||||
fi
|
||||
yarn run prettier "$@"
|
||||
}
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
gofmt -s -w .
|
||||
run_prettier --write '**/*.md' --tab-width 4 --prose-wrap always
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
||||
+2
-21
@@ -1,37 +1,18 @@
|
||||
#!/bin/sh
|
||||
# script/fmt-check: check formatting (read-only). Same scope as
|
||||
# script/fmt: gofmt for Go, prettier for Markdown. Both run every time
|
||||
# and each reports independently, so a failure names which formatter is
|
||||
# unhappy; the script exits non-zero if either found unformatted files.
|
||||
# script/fmt, but fails instead of writing.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
run_prettier() {
|
||||
if ! command -v yarn >/dev/null 2>&1; then
|
||||
echo "fmt-check: yarn not found; run script/bootstrap first" >&2
|
||||
exit 1
|
||||
fi
|
||||
yarn run prettier "$@"
|
||||
}
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
rc=0
|
||||
|
||||
files="$(gofmt -s -l .)"
|
||||
if [ -n "$files" ]; then
|
||||
echo "gofmt: files not formatted:" >&2
|
||||
echo "$files" >&2
|
||||
rc=1
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! run_prettier --check '**/*.md' --tab-width 4 --prose-wrap always; then
|
||||
echo "prettier: Markdown not formatted; run make fmt" >&2
|
||||
rc=1
|
||||
fi
|
||||
|
||||
exit "$rc"
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
||||
@@ -1,8 +0,0 @@
|
||||
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
|
||||
# yarn lockfile v1
|
||||
|
||||
|
||||
prettier@3.8.1:
|
||||
version "3.8.1"
|
||||
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
|
||||
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==
|
||||
Reference in New Issue
Block a user