Author SHA1 Message Date
sneak dd405054fe Refuse an unversioned database that already has a files table (closes #11)
check / check (push) Failing after 1s
An unversioned database (user_version 0) that already contains a files
table was not created by this build; it is a foreign or partially
initialized file. Adopting it silently could corrupt unrelated data, so
createSchema now checks for a files table first and, when one exists,
returns the schema-version error telling the operator to remove the file
and rescan. A genuinely empty database is still created and stamped as
before.

Model: opus-4-8
2026-09-21 22:49:33 +00:00
clawbot 7ac4f6b723 Remove dead files.dat references from build config (closes #22)
check / check (push) Failing after 0s
files.dat was the scan format before the SQLite database; nothing has
produced it since. Drop the stale references from the Makefile clean
target, .gitignore and .dockerignore. make clean still removes the
binary and .gitignore still covers the database files. The only
remaining mention is the historical entry in TODO.md.

Model: opus-4-8 (implementation); fable-5-1 (merge)
2026-09-21 15:01:57 +02:00
15 changed files with 905 additions and 834 deletions
-2
View File
@@ -2,8 +2,6 @@
.claude .claude
.DS_Store .DS_Store
sfdupes sfdupes
files.dat
node_modules
*.log *.log
*.out *.out
*.test *.test
-1
View File
@@ -27,7 +27,6 @@ node_modules/
*.log *.log
# Local scan data # Local scan data
files.dat
*.sqlite *.sqlite
*.sqlite-shm *.sqlite-shm
*.sqlite-wal *.sqlite-wal
-2
View File
@@ -1,2 +0,0 @@
node_modules/
yarn.lock
-4
View File
@@ -1,4 +0,0 @@
{
"tabWidth": 4,
"proseWrap": "always"
}
+6 -11
View File
@@ -27,12 +27,9 @@ ARG CHECK_EPOCH
# target now runs `docker build -f Dockerfile.lint`, and a docker build # target now runs `docker build -f Dockerfile.lint`, and a docker build
# cannot run a docker build: routing the gate through make would mean # cannot run a docker build: routing the gate through make would mean
# nesting docker inside this image. Same reason `make check` is gone # nesting docker inside this image. Same reason `make check` is gone
# from the build stage below. # from the build stage below. `make fmt-check` stays as it is — it is a
# # gate, not the aggregate, and it shells out to nothing.
# `make fmt-check` is not run in this stage: it now also runs prettier RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
# over Markdown, and this golangci-lint image has no node. The gate runs
# in the build stage below, where script/bootstrap installs node and
# prettier.
# The FROM above and the one in Dockerfile.lint pin the same linter # The FROM above and the one in Dockerfile.lint pin the same linter
# twice, and nothing else keeps them in sync; this fails the build when # twice, and nothing else keeps them in sync; this fails the build when
@@ -80,12 +77,10 @@ COPY --from=lint /src/go.sum /dev/null
# rather than duplicating the installs inline. Only script/ and the # rather than duplicating the installs inline. Only script/ and the
# dependency manifests are copied first, nothing else, so this layer # dependency manifests are copied first, nothing else, so this layer
# stays cached until the scripts or the dependencies change — bootstrap # stays cached until the scripts or the dependencies change — bootstrap
# runs `go mod download` and `yarn install`, which is why there is no # ends in `go mod download`, which is why there is no separate
# separate invocation of either here. The JS manifests (package.json, # invocation of it here.
# yarn.lock) are copied too so the yarn install layer caches alongside
# the Go one.
COPY script/ script/ COPY script/ script/
COPY go.mod go.sum package.json yarn.lock ./ COPY go.mod go.sum ./
RUN script/bootstrap RUN script/bootstrap
COPY . . COPY . .
+1 -1
View File
@@ -46,4 +46,4 @@ hooks:
@script/install-precommit @script/install-precommit
clean: clean:
rm -f $(BINARY) files.dat rm -f $(BINARY)
+437 -384
View File
@@ -2,18 +2,19 @@
## Description ## Description
`sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that `sfdupes` is an MIT-licensed Go CLI tool by
quickly identifies _candidate_ duplicate files — and, ultimately, entire [@sneak](https://sneak.berlin) that quickly identifies *candidate*
duplicate directory trees — across very large filesystems without reading full duplicate files — and, ultimately, entire duplicate directory trees —
file contents. Files are considered duplicates when they have identical size, across very large filesystems without reading full file contents. Files
identical SHA-256 of their first 1024 bytes, and identical SHA-256 of their last are considered duplicates when they have identical size, identical
1024 bytes. This is a strong candidate signal, not proof of identical content SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
(the middle of the file is never read); the intended use is finding duplicate 1024 bytes. This is a strong candidate signal, not proof of identical
downloads and duplicated directory trees on multi-terabyte ZFS servers where content (the middle of the file is never read); the intended use is
reading every byte is prohibitively expensive. `scan` maintains a persistent finding duplicate downloads and duplicated directory trees on
SQLite database of file signatures that survives between runs, so it can be run multi-terabyte ZFS servers where reading every byte is prohibitively
from cron and the reports can be generated at any time from the most recent expensive. `scan` maintains a persistent SQLite database of file
scan. signatures that survives between runs, so it can be run from cron and
the reports can be generated at any time from the most recent scan.
This README is the complete and authoritative specification. This README is the complete and authoritative specification.
@@ -27,81 +28,91 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes trees > dupetrees.tsv ./sfdupes trees > dupetrees.tsv
``` ```
`scan` walks one or more filesystem trees and maintains one database record per `scan` walks one or more filesystem trees and maintains one database
regular file (path, size, mtime, head hash, tail hash). The database persists record per regular file (path, size, mtime, head hash, tail hash). The
between runs; a rescan only hashes files that are new or changed, and removes database persists between runs; a rescan only hashes files that are new
records for files that no longer exist. `report` reads the database and prints or changed, and removes records for files that no longer exist.
the file-level duplicates report. `trees` reads the same database and prints the `report` reads the database and prints the file-level duplicates
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation report. `trees` reads the same database and prints the duplicate-tree
with no `PATH` operand — prints a usage message and exits 2. report. A missing/invalid subcommand — or a `scan` invocation with no
`PATH` operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan` anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a
cron job, with the reporting commands run interactively whenever needed; their daily `sfdupes scan` cron job, with the reporting commands run
results are as fresh as the last completed scan. interactively whenever needed; their results are as fresh as the last
completed scan.
## Rationale ## Rationale
Duplicate finders that hash entire files do not scale to the target environment: Duplicate finders that hash entire files do not scale to the target
~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under environment: ~10 million files and ~150 TB on possibly slow or busy
resilver). Reading at most 2 KiB per file — and only from files whose size at disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
least one other file shares, since a size-unique file cannot be a duplicate — only from files whose size at least one other file shares, since a
makes a full-filesystem sweep tractable, and the signatures are kept in a size-unique file cannot be a duplicate — makes a full-filesystem sweep
persistent database, so the expensive filesystem pass is incremental: a rescan tractable, and the signatures are kept in a persistent database, so
re-hashes only files whose recorded mtime or size changed, and all analysis the expensive filesystem pass is incremental: a rescan re-hashes only
happens offline from the database alone. The end goal is not individual files files whose recorded mtime or size changed, and all analysis happens
but whole duplicated trees — duplicate extractions, duplicate downloads, copied offline from the database alone. The end goal is
project trees — which an operator can consider removing as a unit. not individual files but whole duplicated trees — duplicate
extractions, duplicate downloads, copied project trees — which an
operator can consider removing as a unit.
## Design ## Design
Goals, in order: Goals, in order:
1. **Find whole duplicate trees, not just files.** The end goal is to identify 1. **Find whole duplicate trees, not just files.** The end goal is to
places where the exact same set of files and directories exists at two or identify places where the exact same set of files and directories
more paths (duplicate extractions, duplicate downloads, copied project exists at two or more paths (duplicate extractions, duplicate
trees), so the operator can consider removing an entire subtree at once. downloads, copied project trees), so the operator can consider
File-level duplicate detection is the foundation; tree-level detection is removing an entire subtree at once. File-level duplicate detection is
built on top of it. the foundation; tree-level detection is built on top of it.
2. **Never read full file contents.** At most 2 KiB is read per file (first and 2. **Never read full file contents.** At most 2 KiB is read per file
last 1024 bytes), and only files whose size at least one other file shares (first and last 1024 bytes), and only files whose size at least
are read at all — a size-unique file cannot be a duplicate. Scale target: one other file shares are read at all — a size-unique file cannot
tens of millions of files, ~150 TB filesystem, possibly slow or busy disks be a duplicate. Scale target: tens of millions of files, ~150 TB
(ZFS pool under resilver). Holding one small record (path, size, mtime) per filesystem, possibly slow or busy disks (ZFS pool under resilver).
file in memory during a scan is acceptable; holding every file's hashes is Holding one small record (path, size, mtime) per file in memory
not (they stay in the database). during a scan is acceptable; holding every file's hashes is not
3. **Scan incrementally, analyze offline.** The expensive filesystem scan (they stay in the database).
maintains a persistent database; an unchanged file is never read again on a 3. **Scan incrementally, analyze offline.** The expensive filesystem
rescan. All analysis (`report`, `trees`) works from the database alone and scan maintains a persistent database; an unchanged file is never
must never touch the scanned filesystem again. `scan` is designed to be read again on a rescan. All analysis (`report`, `trees`) works from
cronned; the reports run at any time against the last completed scan. the database alone and must never touch the scanned filesystem
4. **Clean stream separation.** Everything on stdout is machine-readable data. again. `scan` is designed to be cronned; the reports run at any
All progress, warnings, and summaries go to stderr. Never mix them. time against the last completed scan.
4. **Clean stream separation.** Everything on stdout is machine-readable
data. All progress, warnings, and summaries go to stderr. Never mix
them.
### Constraints ### Constraints
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`. - Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, **one `sfdupes`.
progress-bar library** (`github.com/schollz/progressbar/v3`), and **one SQLite - Dependencies: standard library, `github.com/spf13/cobra` for the
driver** (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled). CLI, **one progress-bar library**
`github.com/spf13/viper` is permitted if configuration-file support is ever (`github.com/schollz/progressbar/v3`), and **one SQLite driver**
needed, but is not currently used. No other third-party deps. (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
`github.com/spf13/viper` is permitted if configuration-file support
is ever needed, but is not currently used. No other third-party
deps.
- Cross-compilation is not a concern. Builds run with cgo disabled (the - Cross-compilation is not a concern. Builds run with cgo disabled (the
`Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go. `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
- Analysis modes (`report`, `trees`) must be deterministic: identical database - Analysis modes (`report`, `trees`) must be deterministic: identical
contents, identical output, regardless of the order in which records were database contents, identical output, regardless of the order in
inserted. which records were inserted.
### Subcommands ### Subcommands
Three subcommands, all implemented: Three subcommands, all implemented:
1. `scan` — walk the filesystem and synchronize the database: one signature 1. `scan` — walk the filesystem and synchronize the database: one
record per regular file. signature record per regular file.
2. `report` — file-level duplicate report from the database. 2. `report` — file-level duplicate report from the database.
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy 3. `trees` — tree-level duplicate report: reconstruct the directory
from the database records, compute a Merkle-style digest per directory, and hierarchy from the database records, compute a Merkle-style digest
report maximal groups of identical trees. per directory, and report maximal groups of identical trees.
``` ```
sfdupes scan [--workers N] [-x] PATH... sfdupes scan [--workers N] [-x] PATH...
@@ -113,22 +124,25 @@ sfdupes trees > dupetrees.tsv
All three subcommands operate on a single SQLite database file: All three subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable when set - Location: the value of the `SFDUPES_DATABASE` environment variable
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`.
command-line flag. There is no command-line flag.
- `scan` creates the database (and its parent directory) on first use. `report` - `scan` creates the database (and its parent directory) on first
and `trees` require an existing database; a missing database file is a fatal use. `report` and `trees` require an existing database; a missing
error (exit 1) telling the user to run `scan` first. database file is a fatal error (exit 1) telling the user to run
- The database uses WAL journal mode and a busy timeout, so running a report `scan` first.
while a cron `scan` is in progress is safe. The filesystem is authoritative; - The database uses WAL journal mode and a busy timeout, so running a
the database is an eventually-consistent reflection of it. Hashed records are report while a cron `scan` is in progress is safe. The filesystem
committed in batched transactions while the scan is still running (keeping the is authoritative; the database is an eventually-consistent
WAL small and letting concurrent reports observe progress), so a report may reflection of it. Hashed records are committed in batched
see a scan's changes partially applied, and a scan that dies partway leaves a transactions while the scan is still running (keeping the WAL
valid database holding everything hashed so far; the next scan skips those small and letting concurrent reports observe progress), so a
records and converges toward the filesystem. report may see a scan's changes partially applied, and a scan
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database that dies partway leaves a valid database holding everything
with any other version is a fatal error): hashed so far; the next scan skips those records and converges
toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
```sql ```sql
CREATE TABLE files ( CREATE TABLE files (
@@ -140,147 +154,167 @@ All three subcommands operate on a single SQLite database file:
) WITHOUT ROWID; ) WITHOUT ROWID;
``` ```
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed Paths are stored as BLOBs because Unix paths are raw bytes, not
UTF-8. `mtime` is used only for change detection; it is not part of the guaranteed UTF-8. `mtime` is used only for change detection; it is
duplicate key. `head` and `tail` are empty strings when the file has never not part of the duplicate key. `head` and `tail` are empty strings
been hashed because its size was unique as of the last scan that covered it; when the file has never been hashed because its size was unique as
such records still define the file for tree reconstruction but never of the last scan that covered it; such records still define the
participate in duplicate groups. file for tree reconstruction but never participate in duplicate
groups.
### `scan` mode ### `scan` mode
`scan` requires one or more `PATH` operands naming the trees to scan. There is `scan` requires one or more `PATH` operands naming the trees to scan.
no default path; invoking `scan` with no operand is a usage error (usage message There is no default path; invoking `scan` with no operand is a usage
on stderr, exit 2). An operand may be a directory or a regular file; an operand error (usage message on stderr, exit 2). An operand may be a directory
that does not exist is a fatal error (exit 1). Because database records persist or a regular file; an operand that does not exist is a fatal error
between runs and are keyed by absolute path, each operand is resolved to an (exit 1). Because database records persist between runs and are keyed
absolute, lexically cleaned path (symlinks are not resolved) before walking, so by absolute path, each operand is resolved to an absolute, lexically
results do not depend on the working directory. All operands belong to a single cleaned path (symlinks are not resolved) before walking, so results do
scan and are enumerated concurrently: every operand seeds the shared walk worker not depend on the working directory. All operands belong to a single
pool. Overlapping operands are harmless — an operand that duplicates another or scan and are enumerated concurrently: every operand seeds the shared
lies under another is dropped before walking, so every file is reached exactly walk worker pool. Overlapping operands are harmless — an operand that
once and produces one database record. duplicates another or lies under another is dropped before walking,
so every file is reached exactly once and produces one database
record.
`scan` synchronizes the database with the filesystem state under the scanned `scan` synchronizes the database with the filesystem state under the
operands: scanned operands:
- Only a file whose size at least one other file shares is ever read: a - Only a file whose size at least one other file shares is ever
size-unique file cannot be a duplicate, so it is recorded without hashes read: a size-unique file cannot be a duplicate, so it is recorded
(`head` and `tail` empty). The size census covers every file walked this scan without hashes (`head` and `tail` empty). The size census covers
plus every database record outside the scanned operands, so a possible every file walked this scan plus every database record outside
duplicate of a separately scanned tree is still recognized. the scanned operands, so a possible duplicate of a separately
- A file not yet in the database is inserted: hashed when its size is shared, scanned tree is still recognized.
without hashes otherwise. - A file not yet in the database is inserted: hashed when its size
- A file already in the database is **skipped without reading its contents** is shared, without hashes otherwise.
when its lstat size equals the recorded size and its lstat mtime is not newer - A file already in the database is **skipped without reading its
than the recorded mtime. This is what makes a daily rescan cheap. Exception: contents** when its lstat size equals the recorded size and its
an unchanged file whose record lacks hashes is hashed — and its record updated lstat mtime is not newer than the recorded mtime. This is what
— once its size becomes shared, so hashing deferred by size-uniqueness happens makes a daily rescan cheap. Exception: an unchanged file whose
as soon as it could matter. record lacks hashes is hashed — and its record updated — once its
- A file whose mtime is newer than recorded, or whose size differs, is processed size becomes shared, so hashing deferred by size-uniqueness
as if new: re-hashed, or recorded without hashes, per the shared-size rule. happens as soon as it could matter.
- A database record whose path lies under one of the scanned operands but was - A file whose mtime is newer than recorded, or whose size differs,
not successfully processed this run is deleted. This removes records for is processed as if new: re-hashed, or recorded without hashes,
deleted files. It also removes records for paths that failed to stat or hash per the shared-size rule.
this run: the database only ever contains signatures verified by the most - A database record whose path lies under one of the scanned operands
recent scan that covered them (a subsequent successful scan re-adds such but was not successfully processed this run is deleted. This
files). removes records for deleted files. It also removes records for
- Database records outside the scanned operands are untouched, so disjoint trees paths that failed to stat or hash this run: the database only ever
can be scanned on different schedules into the same database. contains signatures verified by the most recent scan that covered
them (a subsequent successful scan re-adds such files).
- Database records outside the scanned operands are untouched, so
disjoint trees can be scanned on different schedules into the same
database.
`scan` runs **three sequential phases over the whole scan**. Parallelism lives `scan` runs **three sequential phases over the whole scan**.
inside each phase; batched database writes begin during the hash phase: Parallelism lives inside each phase; batched database writes begin
during the hash phase:
1. **walk + stat** — enumerate the trees under all `PATH` operands concurrently 1. **walk + stat** — enumerate the trees under all `PATH` operands
with the walk worker pool: every operand seeds the shared queue, and each concurrently with the walk worker pool: every operand seeds the
worker reads one directory at a time, handing discovered subdirectories back shared queue, and each worker reads one directory at a time,
to the queue and running `lstat` on each regular file as it is discovered handing discovered subdirectories back to the queue and running
(while the directory's metadata is still hot). Sequential directory `lstat` on each regular file as it is discovered (while the
enumeration is metadata-latency-bound and takes hours at tens of millions of directory's metadata is still hot). Sequential directory
files; per-directory parallelism is what makes the walk tractable on large enumeration is metadata-latency-bound and takes hours at tens of
or busy pools. The walk builds the size census and resolves unchanged millions of files; per-directory parallelism is what makes the
already-hashed files on the fly; every other file is carried to the hash walk tractable on large or busy pools. The walk builds the size
phase as a (path, size, mtime) record. census and resolves unchanged already-hashed files on the fly;
2. **hash** — with the census complete, each carried file's size decides its every other file is carried to the hash phase as a (path, size,
fate. Size-unique files are never read: new or changed ones are recorded mtime) record.
without hashes in the update phase, unchanged unhashed ones simply keep 2. **hash** — with the census complete, each carried file's size
their records. Every file with a shared size is hashed by the worker pool: decides its fate. Size-unique files are never read: new or
read the first `min(1024, size)` bytes and the last `min(1024, size)` bytes changed ones are recorded without hashes in the update phase,
(one read when `size <= 1024`, since the two windows coincide) and compute unchanged unhashed ones simply keep their records. Every file
the SHA-256 of each. Zero-length files have constant hashes and are never with a shared size is hashed by the worker pool: read the first
opened. Files are hashed in **inode order** (minimizing seeks on spinning `min(1024, size)` bytes and the last `min(1024, size)` bytes
disks), and paths that are hard links to the same inode are **read once**, (one read when `size <= 1024`, since the two windows coincide)
all sharing the one result — a hard-link backup farm costs one read per and compute the SHA-256 of each. Zero-length files have constant
inode, not per path. The phase total counts actual reads, so progress and hashes and are never opened. Files are hashed in **inode order**
ETA are meaningful. Completed records are committed in batched transactions (minimizing seeks on spinning disks), and paths that are hard
**while hashing runs**, so a scan interrupted after hours keeps everything links to the same inode are **read once**, all sharing the one
hashed so far and the next scan resumes cheaply, skipping records already result — a hard-link backup farm costs one read per inode, not
written. per path. The phase total counts actual reads, so progress and
3. **update** — commit the final partial batch, the hash-less records for ETA are meaningful. Completed records are committed in batched
size-unique new and changed files, and the deletions for records the scan transactions **while hashing runs**, so a scan interrupted after
did not verify (vanished files, plus paths that failed to stat or hash). hours keeps everything hashed so far and the next scan resumes
cheaply, skipping records already written.
3. **update** — commit the final partial batch, the hash-less
records for size-unique new and changed files, and the deletions
for records the scan did not verify (vanished files, plus paths
that failed to stat or hash).
Rules for the walk: Rules for the walk:
- Only regular files. Skip directories, symlinks (do not follow, including - Only regular files. Skip directories, symlinks (do not follow,
symlink operands), sockets, FIFOs, and device nodes. including symlink operands), sockets, FIFOs, and device nodes.
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking - Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
them would list every file once per snapshot). walking them would list every file once per snapshot).
- Filesystem boundaries are crossed by default. With `-x` (long form - Filesystem boundaries are crossed by default. With `-x`
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend (long form `--one-file-system`, following the GNU `du`/`rsync`
into a directory on a different filesystem than its `PATH` operand; each convention), never descend into a directory on a different
operand is bounded by its own filesystem. filesystem than its `PATH` operand; each operand is bounded by its
- On any per-path error (permission denied, file vanished between passes, own filesystem.
unreadable): print a one-line warning to stderr, skip the path, and continue. - On any per-path error (permission denied, file vanished between
Per-file errors never abort the run; the final summary reports how many were passes, unreadable): print a one-line warning to stderr, skip the
skipped. As specified above, a skipped path that has a database record from an path, and continue. Per-file errors never abort the run; the final
earlier scan loses that record; an unreadable directory subtree likewise loses summary reports how many were skipped. As specified above, a
its records (accepted: the database mirrors what the latest scan could skipped path that has a database record from an earlier scan loses
that record; an unreadable directory subtree likewise loses its
records (accepted: the database mirrors what the latest scan could
actually verify). actually verify).
Concurrency: the walk phase (which also stats files) and the hash phase each use Concurrency: the walk phase (which also stats files) and the hash
a worker pool of `--workers` workers (default `runtime.NumCPU()`); the walk phase each use a worker pool of `--workers` workers (default
parallelizes across directories, hashing across files. Both phases are `runtime.NumCPU()`); the walk parallelizes across directories,
seek-bound on spinning disks, so raising `--workers` well past the core count hashing across files. Both phases are seek-bound on spinning disks,
can help on pools with many spindles. The main goroutine owns partitioning, so raising `--workers` well past the core count can help on pools
database writes, and progress rendering; progress display must never block the with many spindles. The main goroutine owns partitioning, database
workers. writes, and progress rendering; progress display must never block
the workers.
`scan` writes nothing to stdout. The summary line on stderr reports the files `scan` writes nothing to stdout. The summary line on stderr reports the
seen this run broken down by disposition, plus skips: files seen this run broken down by disposition, plus skips:
``` ```
scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
``` ```
(`removed` counts deleted database records, which are not part of the files-seen (`removed` counts deleted database records, which are not part of the
total.) files-seen total.)
### `report` mode ### `report` mode
`report` reads every record from the database and takes no positional arguments. `report` reads every record from the database and takes no positional
arguments.
**`report` must never touch the filesystem being analyzed.** It does not stat, **`report` must never touch the filesystem being analyzed.** It does not
open, or otherwise access any path that appears in the records; its only I/O is stat, open, or otherwise access any path that appears in the records; its
reading the database and writing stdout/stderr. It must produce identical output only I/O is reading the database and writing stdout/stderr. It must
whether or not the scanned filesystem is still mounted. produce identical output whether or not the scanned filesystem is still
mounted.
Processing: Processing:
- Records without hashes (size-unique when last scanned) are excluded: their - Records without hashes (size-unique when last scanned) are
content is unknown, so they are never reported as duplicates. excluded: their content is unknown, so they are never reported as
- Group the remaining records by the key `(size, head_hash, tail_hash)`. duplicates.
- Group the remaining records by the key
`(size, head_hash, tail_hash)`.
- Every group with two or more paths is a duplicate group. - Every group with two or more paths is a duplicate group.
- Within each group, sort paths lexicographically (byte order). The first path - Within each group, sort paths lexicographically (byte order). The
is the group's `first`; every other path is a `dupe`. first path is the group's `first`; every other path is a `dupe`.
- Order groups by size descending (biggest reclaimable space first), tie-broken - Order groups by size descending (biggest reclaimable space first),
by `first` path ascending. Output must be fully deterministic for a given tie-broken by `first` path ascending. Output must be fully
database state. deterministic for a given database state.
#### Report output format #### Report output format
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a TSV on stdout: a header line, then one row per duplicate file (N-1 rows
group of N): for a group of N):
``` ```
first dupe size first dupe size
@@ -288,86 +322,94 @@ first dupe size
/srv/a/big.iso /srv/c/big-copy2.iso 4294967296 /srv/a/big.iso /srv/c/big-copy2.iso 4294967296
``` ```
Summary to stderr: records read, number of duplicate groups, number of dupe Summary to stderr: records read, number of duplicate groups, number of
files, and total reclaimable bytes (sum of `size` over all dupe rows) in human dupe files, and total reclaimable bytes (sum of `size` over all dupe
units. rows) in human units.
### `trees` mode ### `trees` mode
`trees` reads the same database as `report` (no positional arguments) and `trees` reads the same database as `report` (no positional arguments)
reports **entire duplicate directory trees**: directories under which the exact and reports **entire duplicate directory trees**: directories under
same set of relative paths exists with the exact same file signatures. which the exact same set of relative paths exists with the exact same
file signatures.
**`trees` must never touch the filesystem being analyzed** — the same rule as **`trees` must never touch the filesystem being analyzed** — the same
`report`. The directory hierarchy is reconstructed purely from the paths in the rule as `report`. The directory hierarchy is reconstructed purely from
records, split on `/`. the paths in the records, split on `/`.
Definitions: Definitions:
- A file's **signature** is `(size, head_hash, tail_hash)` — mtime is - A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
informational and excluded. An unhashed record (empty hashes) has unknown informational and excluded. An unhashed record (empty hashes) has
content: its signature is treated as unique to that file, so a tree containing unknown content: its signature is treated as unique to that file,
an unhashed file never compares equal to any other tree. so a tree containing an unhashed file never compares equal to any
- A directory's **digest** is a SHA-256 Merkle digest computed bottom-up: other tree.
serialize the directory's child entries — for a file child, its name and - A directory's **digest** is a SHA-256 Merkle digest computed
signature; for a subdirectory child, its name and that subdirectory's digest — bottom-up: serialize the directory's child entries — for a file
sort the serialized entries byte-lexicographically, and hash the child, its name and signature; for a subdirectory child, its name
concatenation. Names are part of the digest: two trees whose files differ only and that subdirectory's digest — sort the serialized entries
in name are _not_ duplicates. byte-lexicographically, and hash the concatenation. Names are part
- Two directories are **duplicate trees** when their digests are equal. Equal of the digest: two trees whose files differ only in name are *not*
digests imply equal recursive file count and equal total byte size. duplicates.
- Two directories are **duplicate trees** when their digests are
equal. Equal digests imply equal recursive file count and equal
total byte size.
Known limitation (accepted): hard-linked paths are reported as duplicates by Known limitation (accepted): hard-linked paths are reported as
`report` and count toward duplicate trees — their content is genuinely identical duplicates by `report` and count toward duplicate trees — their
— even though they share storage, so removing one reclaims no space. Inode content is genuinely identical — even though they share storage, so
identity is used during the scan to avoid redundant reads but is not persisted removing one reclaims no space. Inode identity is used during the
in the database. scan to avoid redundant reads but is not persisted in the database.
Known limitation (accepted): only regular files that appear in the database Known limitation (accepted): only regular files that appear in the
define a tree. Empty directories are invisible, and a file skipped during the database define a tree. Empty directories are invisible, and a file
scan (e.g. permission error) in one copy but not the other will make skipped during the scan (e.g. permission error) in one copy but not the
otherwise-identical trees compare as different. other will make otherwise-identical trees compare as different.
Processing: Processing:
- Build the hierarchy, compute every directory's digest, and group directories - Build the hierarchy, compute every directory's digest, and group
by digest. Every group with two or more directories is a duplicate-tree group. directories by digest. Every group with two or more directories is a
- **Report only maximal trees.** A group is suppressed when its members' parents duplicate-tree group.
are pairwise distinct directories that all share a single digest — such a - **Report only maximal trees.** A group is suppressed when its
group is wholly implied by its parents' (or a further ancestor's) group. members' parents are pairwise distinct directories that all share a
Groups containing sibling directories, or members whose parents differ, are single digest — such a group is wholly implied by its parents' (or a
always reported. further ancestor's) group. Groups containing sibling directories, or
- Within each group, sort paths lexicographically (byte order); the first path members whose parents differ, are always reported.
is `first`, every other path is a `dupe`. - Within each group, sort paths lexicographically (byte order); the
- Order groups by total tree size descending, tie-broken by `first` path first path is `first`, every other path is a `dupe`.
ascending. Output must be fully deterministic for a given input. - Order groups by total tree size descending, tie-broken by `first`
path ascending. Output must be fully deterministic for a given
input.
#### Trees output format #### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a TSV on stdout: a header line, then one row per duplicate tree (N-1 rows
group of N). `files` is the recursive regular-file count of one copy of the for a group of N). `files` is the recursive regular-file count of one
tree; `size` is the recursive total byte size of one copy: copy of the tree; `size` is the recursive total byte size of one copy:
``` ```
first dupe files size first dupe files size
/srv/a/project /srv/backup/project 3417 104857600 /srv/a/project /srv/backup/project 3417 104857600
``` ```
Summary to stderr: records read, number of duplicate-tree groups, number of dupe Summary to stderr: records read, number of duplicate-tree groups,
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human number of dupe trees, and total reclaimable bytes (sum of `size` over
units. all dupe rows) in human units.
### Progress ### Progress
Use the progress-bar library for all scan progress; rendering in the style of Use the progress-bar library for all scan progress; rendering in the
`pv` is the model. All progress goes to stderr. style of `pv` is the model. All progress goes to stderr.
Each phase gets its own display, rendered the moment the phase starts — a scan Each phase gets its own display, rendered the moment the phase
must never look hung. Loading the existing-record index (`load`) and the walk starts — a scan must never look hung. Loading the existing-record
have no known totals while running: show a live count, rate, and elapsed time index (`load`) and the walk have no known totals while running: show
(spinner-style, no percentage or ETA). The hash and update phases have exact a live count, rate, and elapsed time (spinner-style, no percentage or
totals — only files that actually need hashing appear in the hash total, so its ETA). The hash and update phases
ETA is meaningful. Required elements for the bars with known totals: have exact totals — only files that actually need hashing appear in
the hash total, so its ETA is meaningful. Required elements for the
bars with known totals:
- elapsed time - elapsed time
- estimated time remaining - estimated time remaining
@@ -382,136 +424,145 @@ hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
Additional requirements: Additional requirements:
- When stderr is not a TTY, do not emit ANSI redraws: print a plain one-line - When stderr is not a TTY, do not emit ANSI redraws: print a plain
progress update no more often than every 5 seconds instead. one-line progress update no more often than every 5 seconds instead.
- Progress updates are driven from the main goroutine and must be non-blocking - Progress updates are driven from the main goroutine and must be
with respect to the worker pool. non-blocking with respect to the worker pool.
- `report` and `trees` modes need no progress display, only their stderr - `report` and `trees` modes need no progress display, only their
summaries. stderr summaries.
### Error handling and exit codes ### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings. - `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., a `PATH` operand does not exist, the database cannot - `1`: fatal error (e.g., a `PATH` operand does not exist, the
be created/opened/read/written, a missing database for `report`/`trees`, database cannot be created/opened/read/written, a missing database
stdout write failure). for `report`/`trees`, stdout write failure).
- `2`: usage error (including `scan` with no `PATH` operand and `report`/`trees` - `2`: usage error (including `scan` with no `PATH` operand and
with any positional argument). `report`/`trees` with any positional argument).
## Entrypoints ## Entrypoints
This repository adheres to the This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: the normalized executables in `script/` are the entrypoints for the standard: the normalized executables in `script/` are the entrypoints
development workflow, and the `Makefile` targets are thin shims that call them. for the development workflow, and the `Makefile` targets are thin
Every script is POSIX `sh`, resolves the repository root itself so it can be run shims that call them. Every script is POSIX `sh`, resolves the
from any working directory, and may be invoked directly. The provided repository root itself so it can be run from any working directory,
entrypoints are: and may be invoked directly. The provided entrypoints are:
- `script/bootstrap` — install everything needed to build and develop this - `script/bootstrap` — install everything needed to build and
repository, idempotently, assuming nothing is present. `git`, `make`, `go`, develop this repository, idempotently, assuming nothing is
and `node` come from the first of nix, apt, brew, or apk found on the host, present. `git`, `make`, and `go` come from the first of nix, apt,
and are presence-checked only; `node` is an unpinned host runtime like the brew, or apk found on the host, and are presence-checked only.
rest, because nvm's prebuilt node is glibc-linked and does not run on this `golangci-lint` is deliberately **not** installed: it runs from a
repo's musl/Alpine build image. The Markdown formatter itself — `prettier` — digest-pinned image via `script/lint` and never from a host
is pinned by `yarn.lock`'s integrity hash and installed with install, so there is no host copy to drift from the pin. A missing
`yarn install --frozen-lockfile`. `golangci-lint` is deliberately **not** `docker` is warned about rather than installed or treated as
installed: it runs from a digest-pinned image via `script/lint` and never from fatal — everything except linting works without it. Ends with
a host install, so there is no host copy to drift from the pin. A missing `go mod download`.
`docker` is warned about rather than installed or treated as fatal —
everything except linting works without it. Ends with `go mod download` and
the `yarn` install.
- `script/setup` — make a fresh clone ready for development: runs - `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`. `script/bootstrap`, then `script/install-precommit`.
- `script/projectname` — print this project's name (`sfdupes`). Scripts that - `script/projectname` — print this project's name (`sfdupes`).
need the name call it, so they stay identical across repositories. Scripts that need the name call it, so they stay identical across
- `script/test` — run the test suite with a 30-second timeout and coverage repositories.
enabled, rerunning verbosely on failure so the logs show which test failed. - `script/test` — run the test suite with a 30-second timeout and
- `script/lint` — run the linter. It builds `Dockerfile.lint`, which copies the coverage enabled, rerunning verbosely on failure so the logs show
repository into the digest-pinned `golangci/golangci-lint` image and runs which test failed.
`golangci-lint config verify` and `golangci-lint run` as build steps, so a - `script/lint` — run the linter. It builds `Dockerfile.lint`, which
successful build is a clean lint. The linter is never run on the host, which copies the repository into the digest-pinned
makes a working `docker` the one prerequisite for linting — and therefore for `golangci/golangci-lint` image and runs
`make check` and the pre-commit hook. Offline machines: the gate steps `golangci-lint config verify` and `golangci-lint run` as build
themselves make no network calls. `golangci-lint run` does not, and neither steps, so a successful build is a clean lint. The linter is never
does `golangci-lint config verify` — it validates against a schema the pinned run on the host, which makes a working `docker` the one
binary embeds, measured under `--network none` to both pass a valid config and prerequisite for linting — and therefore for `make check` and the
reject an invalid one. The build around them does. `Dockerfile.lint` runs pre-commit hook. Offline machines: the gate steps themselves make
`go mod download` before the gates and this module has external dependencies, no network calls. `golangci-lint run` does not, and neither does
so a first lint on a machine with a cold BuildKit cache reaches the network `golangci-lint config verify` — it validates against a schema the
there (as well as pulling the pinned image); under `--network none` it fails pinned binary embeds, measured under `--network none` to both
at that step, before any gate. That layer sits above the gates and stays pass a valid config and reject an invalid one. The build around
cached, so once it is warm `script/lint` — and with it `make check` — runs them does. `Dockerfile.lint` runs `go mod download` before the
entirely offline, until `go.mod` or `go.sum` changes and the download layer gates and this module has external dependencies, so a first lint
goes cold again. Because the daemon only ever sees a build context, this works on a machine with a cold BuildKit cache reaches the network there
when the docker daemon is remote and bind mounts are impossible. (as well as pulling the pinned image); under `--network none` it
- `script/fmt` — format in place: `gofmt -s -w` for Go sources and `prettier` fails at that step, before any gate. That layer sits above the
for Markdown (`--tab-width 4 --prose-wrap always`, the house settings, also gates and stays cached, so once it is warm `script/lint` — and
carried in `.prettierrc`). prettier is the pinned devDependency in with it `make check` — runs entirely offline, until `go.mod` or
`package.json`/`yarn.lock`, installed by `script/bootstrap`. `go.sum` changes and the download layer goes cold again. Because
- `script/fmt-check` — the read-only counterpart of `script/fmt`: runs both the daemon only ever sees a build context, this works when the
checks, reports each independently so it is clear which failed, and exits docker daemon is remote and bind mounts are impossible.
non-zero if either found unformatted files instead of writing. - `script/fmt` — format the Go sources in place (`gofmt -s -w`).
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in Markdown is not formatted.
that order. Modifies nothing. Needs `docker`, because `script/lint` does. - `script/fmt-check` — the read-only counterpart of `script/fmt`:
- `script/docker` — build the Docker image, tagged with the name from prints any unformatted file and exits non-zero instead of writing.
`script/projectname`. The `Dockerfile` runs the gates as build steps, so this - `script/check` — run `script/test`, `script/lint`, and
is also the check a developer or reviewer runs by hand. `script/fmt-check`, in that order. Modifies nothing. Needs
- `script/cibuild` — build the Docker image untagged. This is what the Gitea `docker`, because `script/lint` does.
workflow runs on push; because the gates run as build steps, a successful - `script/docker` — build the Docker image, tagged with the name
build implies the repository is green. from `script/projectname`. The `Dockerfile` runs the gates as
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a build steps, so this is also the check a developer or reviewer
no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then runs by hand.
`script/check`. - `script/cibuild` — build the Docker image untagged. This is what
- `script/install-precommit` — install the git pre-commit hook that runs the Gitea workflow runs on push; because the gates run as build
`script/precommit`. The hook is written to the common git directory, so the steps, a successful build implies the repository is green.
main checkout and every worktree share it. - `script/precommit` — run by the git pre-commit hook: `go mod tidy`
- `script/verify-lint-image-pin` — fail unless the `golangci/golangci-lint` must be a no-op (a resulting change to `go.mod` or `go.sum` fails
reference in `Dockerfile.lint` and the one in the `Dockerfile` lint stage are the commit), then `script/check`.
the same image at the same digest, naming both if not. The linter is pinned in - `script/install-precommit` — install the git pre-commit hook that
those two files and nothing else keeps them in sync, so a bump applied to one runs `script/precommit`. The hook is written to the common git
alone would leave `make lint` and the `Dockerfile`'s fail-fast lint stage directory, so the main checkout and every worktree share it.
checking the same tree against different rulesets, both green. The guard - `script/verify-lint-image-pin` — fail unless the
restates neither pin — a third copy would be the same drift one file further `golangci/golangci-lint` reference in `Dockerfile.lint` and the
out — and runs as a gate in both files, so `make lint`, `make check` and one in the `Dockerfile` lint stage are the same image at the same
`make docker` all catch it. digest, naming both if not. The linter is pinned in those two
files and nothing else keeps them in sync, so a bump applied to
one alone would leave `make lint` and the `Dockerfile`'s
fail-fast lint stage checking the same tree against different
rulesets, both green. The guard restates neither pin — a third
copy would be the same drift one file further out — and runs as a
gate in both files, so `make lint`, `make check` and `make docker`
all catch it.
`script/verify-linter-pin` used to live here. It compared a linter binary `script/verify-linter-pin` used to live here. It compared a linter
against a version pin in `script/bootstrap`, and both of its subjects are gone: binary against a version pin in `script/bootstrap`, and both of its
no linter binary is copied between build stages any more, and bootstrap pins no subjects are gone: no linter binary is copied between build stages any
version because it installs no linter. The drift it existed to catch has moved more, and bootstrap pins no version because it installs no linter. The
from binary-versus-pin to pin-versus-pin, which is what drift it existed to catch has moved from binary-versus-pin to
`script/verify-lint-image-pin` above checks. pin-versus-pin, which is what `script/verify-lint-image-pin` above
checks.
`script/lint`, `script/docker` and `script/cibuild` all pass a freshly computed `script/lint`, `script/docker` and `script/cibuild` all pass a freshly
`CHECK_EPOCH` build argument, and the gate steps in `Dockerfile.lint` and computed `CHECK_EPOCH` build argument, and the gate steps in
`Dockerfile` reference it. Without that, an unchanged tree lets Docker serve the `Dockerfile.lint` and `Dockerfile` reference it. Without that, an
gate layers from cache and the build exits 0 having executed no tests and no unchanged tree lets Docker serve the gate layers from cache and the
lint — a green it never earned, and one this repository has produced twice. build exits 0 having executed no tests and no lint — a green it never
`CHECK_EPOCH` invalidates the gate layers on every run while leaving the pinned earned, and one this repository has produced twice. `CHECK_EPOCH`
base images and the dependency layers cached. `script/lint`'s value carries the invalidates the gate layers on every run while leaving the pinned base
process id as well as the epoch, because two lint runs land inside the same images and the dependency layers cached. `script/lint`'s value carries
second easily and a bare epoch would cache the second one. the process id as well as the epoch, because two lint runs land inside
the same second easily and a bare epoch would cache the second one.
## Build ## Build
The `script/` entrypoints above are where the implementations live; the The `script/` entrypoints above are where the implementations live;
`Makefile` targets are shims onto them, except `build`, which carries the the `Makefile` targets are shims onto them, except `build`, which
compile recipe: carries the compile recipe:
- `make` / `make build` — build the `sfdupes` binary (cgo disabled); building is - `make` / `make build` — build the `sfdupes` binary (cgo
the default target. disabled); building is the default target.
- `make bootstrap` — install the build and development dependencies. - `make bootstrap` — install the build and development
- `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook. dependencies.
- `make test` — run the test suite (30-second timeout; reruns with `-v` on - `make setup` — prepare a fresh clone: `bootstrap` plus the
failure). pre-commit hook.
- `make lint` — run `golangci-lint` with the repo config, in Docker (see - `make test` — run the test suite (30-second timeout; reruns with
`script/lint`); requires `docker`. `-v` on failure).
- `make fmt` / `make fmt-check` — format Go and Markdown sources / verify both - `make lint` — run `golangci-lint` with the repo config, in Docker
without writing. (see `script/lint`); requires `docker`.
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires - `make fmt` / `make fmt-check` — format Go sources / verify
`docker`, via `lint`. formatting without writing.
- `make docker` — build the Docker image, which runs the gates as build stages. - `make check` — `test`, `lint`, and `fmt-check`; modifies nothing.
Requires `docker`, via `lint`.
- `make docker` — build the Docker image, which runs the gates as
build stages.
- `make hooks` — install the pre-commit hook. - `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary. - `make clean` — remove the binary.
@@ -521,8 +572,8 @@ All of the following, run in this directory, must pass:
1. `make check` passes (tests, lint, `gofmt`). 1. `make check` passes (tests, lint, `gofmt`).
2. `make docker` succeeds. 2. `make docker` succeeds.
3. Smoke test — create a throwaway tree in a temp dir (never test against real 3. Smoke test — create a throwaway tree in a temp dir (never test
data): against real data):
```sh ```sh
d=$(mktemp -d) d=$(mktemp -d)
@@ -554,29 +605,30 @@ All of the following, run in this directory, must pass:
./sfdupes report ./sfdupes report
``` ```
(The scan database lives inside `$d` here purely for test hygiene; scanning (The scan database lives inside `$d` here purely for test hygiene;
`$d` therefore also records the SQLite file itself, which is harmless.) scanning `$d` therefore also records the SQLite file itself, which
is harmless.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin`
group (two dupe rows, `first` is the lexicographically smallest path); form one group (two dupe rows, `first` is the lexicographically
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/ smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group;
`t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair; `t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group;
`empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; groups `tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and
ordered by size descending. `tiny3` appear nowhere; groups ordered by size descending.
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2 Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal `$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is
(implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
differs by name). appears nowhere (its file set differs by name).
Expected from the second `report` (after the modify/delete rescan): Expected from the second `report` (after the modify/delete rescan):
`one.bin` has left its group (its content changed), so `one.bin` has left its group (its content changed), so
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is gone from the `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is
database. gone from the database.
The test suite automates this scenario (see `scan_test.go`), plus a negative The test suite automates this scenario (see `scan_test.go`), plus a
check: `report` and `trees` operate on the database alone and never touch negative check: `report` and `trees` operate on the database alone
the scanned filesystem. and never touch the scanned filesystem.
## TODO ## TODO
@@ -584,10 +636,11 @@ Tracked in [TODO.md](TODO.md).
## Non-goals ## Non-goals
- No full-content verification, no byte-for-byte compare, no deletion or linking - No full-content verification, no byte-for-byte compare, no deletion
of duplicates. The reports are advisory; acting on them is the user's job. or linking of duplicates. The reports are advisory; acting on them is
- No persistence beyond the SQLite database described above; no export/import the user's job.
formats. - No persistence beyond the SQLite database described above; no
export/import formats.
- No daemon or filesystem watcher; scheduling rescans is cron's job. - No daemon or filesystem watcher; scheduling rescans is cron's job.
## License ## License
+388 -335
View File
@@ -1,386 +1,439 @@
# Workflow # Workflow
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the - take an issue from the `1.0.0` milestone on the tracker; work not
tracker gets filed as an issue first yet on the tracker gets filed as an issue first
- branch (from `main`) - branch (from `main`)
- do the work, with tests, in small focused commits - do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the same commit - record it at the top of Completed Steps (`TODO.md` changes in the
as the work) same commit as the work)
- push the branch and open a PR whose title ends with ` (closes #N)` - push the branch and open a PR whose title ends with
- an independent review gates the merge; every finding is addressed or ` (closes #N)`
explicitly rebutted on the PR - an independent review gates the merge; every finding is addressed
or explicitly rebutted on the PR
- merge to `main` once the review passes - merge to `main` once the review passes
# Status # Status
- pre-1.0 - pre-1.0
- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues - the Gitea tracker is authoritative for the pre-1.0 backlog: the
under the `1.0.0` milestone are what remains before the tag, and this file open issues under the `1.0.0` milestone are what remains before
records history and process, not the queue the tag, and this file records history and process, not the queue
# Next Step # Next Step
- take the next issue from the `1.0.0` milestone on the tracker: - take the next issue from the `1.0.0` milestone on the tracker:
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
of truth for what is left before 1.0.0. Individual issues are deliberately not the source of truth for what is left before 1.0.0. Individual
restated here; a copy in this file drifts out of date the moment the tracker issues are deliberately not restated here; a copy in this file
moves drifts out of date the moment the tracker moves
# Completed Steps # Completed Steps
- restore Markdown formatting in `script/fmt`/`fmt-check` and reformat all - refuse an unversioned database that already has a `files` table with
Markdown to the house prettier settings (2026-09-21, closes a clear schema-version error (2026-09-21, closes
https://git.eeqj.de/sneak/sfdupes/issues/19) https://git.eeqj.de/sneak/sfdupes/issues/11)
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
and `.dockerignore` (2026-09-21, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/22)
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and - fix the lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` (2026-08-10, branch `next`, closes `Dockerfile.lint` (2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
`(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag, `(Debian-based)` parenthetical (v2.12.1 was Debian too) and the
so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare redundant tag, so both pins are the policy `# image:vX.Y.Z,
`FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin` YYYY-MM-DD` comment over a bare `FROM image@sha256:...`. Digest
parses those `FROM` lines and still matches the tagless form; its advice line unchanged. `script/verify-lint-image-pin` parses those `FROM` lines
lost the now meaningless "tag and digest". With no tag in either reference, a and still matches the tagless form; its advice line lost the now
tag-only disagreement no longer exists — a one-sided tag is caught as a plain meaningless "tag and digest". With no tag in either reference, a
mismatch. tag-only disagreement no longer exists — a one-sided tag is caught as
a plain mismatch.
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10, - run all linting in Docker via `Dockerfile.lint` and `script/lint`
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the (2026-08-10, branch `next`, closes
owner ruling, the linter runs inside a container invoked through the `script/` https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling, the
entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs linter runs inside a container invoked through the `script/`
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and entrypoint and is never installed on a host. New root
runs `golangci-lint config verify` and `golangci-lint run` as build steps, so `Dockerfile.lint` COPYs the repo into the digest-pinned
a successful build IS a clean lint; `script/lint` is reduced to building it. `golangci/golangci-lint:v2.12.2` image and runs
`script/bootstrap` loses the `go install`, the pin constants, the version `golangci-lint config verify` and `golangci-lint run` as build
parser and `verify_golangci_lint` outright rather than hardening them — with steps, so a successful build IS a clean lint; `script/lint` is
nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that reduced to building it. `script/bootstrap` loses the `go install`,
motivated them has no subject — and now warns rather than fails when `docker` the pin constants, the version parser and `verify_golangci_lint`
is absent. Two traps handled. A lint build on an unchanged tree returns outright rather than hardening them — with nothing linting on the
success in well under a second having run no linter, which is host, the `$GOPATH/bin` versus `PATH` problem that motivated them has
no subject — and now warns rather than fails when `docker` is absent.
Two traps handled. A lint build on an unchanged tree returns success
in well under a second having run no linter, which is
https://git.eeqj.de/sneak/sfdupes/issues/32 and https://git.eeqj.de/sneak/sfdupes/issues/32 and
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint` https://git.eeqj.de/sneak/sfdupes/issues/39 again, so
carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes `Dockerfile.lint` carries `ARG CHECK_EPOCH` referenced
the expanded command, not the declaration) and `script/lint` passes inside every gate `RUN` (BuildKit hashes the expanded command, not
`"$(date +%s)-$$"` the PID matters because two lint runs land inside the the declaration) and `script/lint` passes `"$(date +%s)-$$"` — the
same second easily. And nothing inside an image build may shell out to docker, PID matters because two lint runs land inside the same second easily.
so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly And nothing inside an image build may shell out to docker, so the
main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
instead of `make lint`, and its build stage runs `make test` and instead of `make lint`, and its build stage runs `make test` and
`make fmt-check` instead of the `make check` aggregate (`make`, not the `make fmt-check` instead of the `make check` aggregate (`make`, not
scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches the scripts bare, because the Makefile's `export CGO_ENABLED = 0`
what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by only reaches what it invokes). `COPY --from=lint`
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge `/usr/bin/golangci-lint` is replaced by
forcing BuildKit to finish linting before the build stage starts, and dropping `COPY --from=lint /src/go.sum /dev/null`: the copied binary was the
it without replacing the edge would have ended fail-fast linting silently only edge forcing BuildKit to finish linting before the build stage
under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering starts, and dropping it without replacing the edge would have ended
edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the fail-fast linting silently under a still-green build. That is
`go install` that justified it. `script/verify-linter-pin` is retired, deleted canonical `REPO_POLICIES.md:107`'s ordering edge, restored.
along with its README entry, because both of its subjects ceased to exist in `ENV PATH=/home/builder/go/bin:$PATH` is gone with the `go install`
the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION` that justified it. `script/verify-linter-pin` is retired, deleted
in `script/bootstrap`, and there is now neither a binary crossing between along with its README entry, because both of its subjects ceased to
stages nor a version pin in bootstrap. The drift it guarded has not gone away, exist in the same change: it compared a linter binary against
it has moved — the linter is still pinned twice, now as the `FROM` line of `GOLANGCI_LINT_VERSION` in `script/bootstrap`, and there is now
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with neither a binary crossing between stages nor a version pin in
nothing syncing them, which is exactly what bootstrap. The drift it guarded has not gone away, it has moved — the
linter is still pinned twice, now as the `FROM` line of
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage,
with nothing syncing them, which is exactly what
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
replacement is one new `script/verify-lint-image-pin`, run as a gate in both replacement is one new `script/verify-lint-image-pin`,
files, which compares the two references to each other and deliberately run as a gate in both files, which compares the two references to
restates neither: a hardcoded expected digest would be a third copy and the each other and deliberately restates neither: a hardcoded expected
same drift one file further out. `golangci-lint config verify` is included per digest would be a third copy and the same drift one file further out.
the ruling, and the concern about its unpinned live HTTPS schema fetch was `golangci-lint config verify` is included per the ruling, and the
measured rather than assumed — under `--network none` the pinned binary both concern about its unpinned live HTTPS schema fetch was measured
passes a valid config and rejects an invalid one with the jsonschema error, so rather than assumed — under `--network none` the pinned binary both
it validates from an embedded schema and makes no network call of its own. The passes a valid config and rejects an invalid one with the jsonschema
README scopes that to the gate steps rather than to linting as a whole: error, so it validates from an embedded schema and makes no network
`Dockerfile.lint` runs `go mod download` above them, so a cold cache still call of its own. The README scopes that to the gate steps rather
needs the network and only a warm one lints offline. Verified: `make lint` than to linting as a whole: `Dockerfile.lint` runs `go mod download`
green with every `PATH` directory containing a `golangci-lint` removed above them, so a cold cache still needs the network and only a warm
one lints offline. Verified: `make lint` green with every `PATH`
directory containing a `golangci-lint` removed
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`; (`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
`command -v golangci-lint` empty); two consecutive `script/lint` runs on an `command -v golangci-lint` empty); two consecutive `script/lint` runs
untouched tree both executed the linter, 27.7s and 28.7s in the lint step on an untouched tree both executed the linter, 27.7s and 28.7s in the
under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s lint step under distinct epochs with the `COPY . .` layer `CACHED`
and 41.8s wall clock — the no-cache rule was not weakened to shorten that. above them, at 42.2s and 41.8s wall clock — the no-cache rule was not
Negative control: a planted `var unusedIssue46Sentinel = 1` failed weakened to shorten that. Negative control: a planted
`script/lint` with `var unusedIssue46Sentinel = 1` failed `script/lint` with
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed `report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and
`make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]` failed `make docker` at `[lint 9/9]` with the build stage stopped at
`COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all `[builder 3/12]``COPY --from=lint`, `script/bootstrap`, the test
zero occurrences — then reverted clean. The drift guard fails on a tag-only gate and `make build` all zero occurrences — then reverted clean. The
disagreement, on a digest-only disagreement, and on an unreadable reference, drift guard fails on a tag-only disagreement, on a digest-only
naming both sides. `make docker` green in 5m35s with all six gates executing disagreement, and on an unreadable reference, naming both sides.
under one epoch (lint 37.6s, test 25.2s reporting `make docker` green in 5m35s with all six gates executing under one
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The epoch (lint 37.6s, test 25.2s reporting
non-root quirk still holds: in the builder image with the Go test cache off, `ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`).
`--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the The non-root quirk still holds: in the builder image with the Go test
unprivileged user passes (exit 0). Noted for follow-up, not fixed here: cache off, `--user 0:0` fails `TestScanHardlinkRunFailsTogether`
`golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0 (exit 1) where the unprivileged user passes (exit 0). Noted for
in favour of `gomodguard_v2`. follow-up, not fixed here: `golangci-lint` warns that the
`gomodguard` linter is deprecated since v2.12.0 in favour of
`gomodguard_v2`.
- install the Docker build stage's prerequisites by running `script/bootstrap` - install the Docker build stage's prerequisites by running
instead of `apk add --no-cache make` inline (2026-08-09, branch `script/bootstrap` instead of `apk add --no-cache make` inline
`dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires (2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical
it, and the inline install left the build stage maintaining its own notion of `REPO_POLICIES.md:97` requires it, and the inline install left the
the toolchain — exactly the divergence #24 exists to close, one layer down. build stage maintaining its own notion of the toolchain — exactly
The stage now copies `script/` plus `go.mod`/`go.sum` and runs the divergence #24 exists to close, one layer down. The stage now
`script/bootstrap`, which ends in `go mod download`, so the separate copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, which ends in `go mod download`, so the separate invocation of that
and moves above the bootstrap layer. It is the only edge making this stage is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves
depend on the lint stage, so deleting it as redundant would end fail-fast above the bootstrap layer. It is the only edge making this stage
linting silently. Letting bootstrap install its own linter here would have depend on the lint stage, so deleting it as redundant would end
reintroduced the second toolchain and paid for a from-source build of it. What fail-fast linting silently. Letting bootstrap install its own linter
makes the two stages provably one toolchain rather than two that happen to here would have reintroduced the second toolchain and paid for a
agree is a new `script/verify-linter-pin`, run in the build stage on the from-source build of it. What makes the two stages provably one
binary that arrives from the lint stage, before bootstrap: it fails the build toolchain rather than two that happen to agree is a new
naming both versions unless that binary is the version `script/bootstrap` `script/verify-linter-pin`, run in the build stage on the binary
pins. Bootstrap's own check could not serve that purpose — it reinstalls its that arrives from the lint stage, before bootstrap: it fails the
pin from source and then verifies whatever `PATH` resolves, so drift build naming both versions unless that binary is the version
self-heals silently and a lint stage image bumped on its own would lint at the `script/bootstrap` pins. Bootstrap's own check could not serve that
new version while `make check` ran at the old one, green. The linter version purpose — it reinstalls its pin from source and then verifies
is pinned in two independent places (the lint stage image digest and whatever `PATH` resolves, so drift self-heals silently and a lint
stage image bumped on its own would lint at the new version while
`make check` ran at the old one, green. The linter version is pinned
in two independent places (the lint stage image digest and
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a `GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
half-applied bump is now a build failure. The pin is read out of half-applied bump is now a build failure. The pin is read out of
`script/bootstrap`, which stays the single source of truth; a pin that cannot `script/bootstrap`, which stays the single source of truth; a pin
be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its that cannot be read is a hard failure, not a skip. The check needs
only inputs are the copied binary and `script/`, so Docker invalidates the no `CHECK_EPOCH`: its only inputs are the copied binary and
layer exactly when a cached result would stop being true, and it is documented `script/`, so Docker invalidates the layer exactly when a cached
with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because result would stop being true, and it is documented with the other
that is where bootstrap's `go install` lands and bootstrap verifies its entrypoints in the README. `$GOPATH/bin` joins `PATH` because
installs against what `PATH` resolves — nothing in the image is shadowed by that is where bootstrap's `go install` lands and bootstrap verifies
it, the directory does not exist until bootstrap runs. Everything added sits its installs against what `PATH` resolves — nothing in the image is
above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede shadowed by it, the directory does not exist until bootstrap runs.
`make check`. Verified: the guard fails the build with both versions named Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and
when the lint stage's linter is faked to a different version, and an `USER builder` still precede `make check`. Verified: the guard fails
unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and the build with both versions named when the lint stage's linter is
its `apk` branch, installing `git` and `make` and finding the copied linter faked to a different version, and an unmodified build still passes
already at the pin; a second build served the bootstrap and dependency layers it; bootstrap runs clean under Alpine's `sh` and its `apk` branch,
`CACHED` while both gates ran with a fresh epoch; a planted `unused` finding installing `git` and `make` and finding the copied
failed the build at the lint gate in 48.9s with the build stage's `make check` linter already at the pin; a second build served the bootstrap and
never starting; and the suite run in the image as `--user 0:0` fails dependency layers `CACHED` while both gates ran with a fresh epoch;
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is a planted `unused` finding failed the build at the lint gate in
still load-bearing. That last check needs the Go test cache disabled — the 48.9s with the build stage's `make check` never starting; and the
first attempt reported `ok ... (cached)` as root, reusing the result the suite run in the image as `--user 0:0` fails
build-time run had left in the shared cache, which would have read as a pass. `TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged
Build wall time, on a shared host running many concurrent builds and so noisy: user is still load-bearing. That last check needs the Go test cache
2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source disabled — the first attempt reported `ok ... (cached)` as root,
change, 5m14s cold. Only the cold one breaches the policy ceiling, and not reusing the result the build-time run had left in the shared cache,
because of this change — `chown -R builder:builder /src /home/builder` walks which would have read as a pass. Build wall time, on a shared host
the module cache and re-runs on every source change, and it alone varied running many concurrent builds and so noisy: 2m13s on an unchanged
between 77s and 210s across those four builds, which is also the whole spread tree, 2m17s and 4m29s for two builds after a source change, 5m14s
in the totals. The same cold measurement against `main` is 5m03s with a 209s cold. Only the cold one breaches the policy ceiling, and not because
`chown`. Filed as #43 of this change — `chown -R builder:builder /src /home/builder` walks
- bust the Docker layer cache for the gate steps, so `script/cibuild` and the module cache and re-runs on every source change, and it alone
`script/docker` cannot report a green they did not earn (2026-08-09, branch varied between 77s and 210s across those four builds, which is also
`cibuild-cache-bust`, closes #32): both scripts were bare `docker build` the whole spread in the totals. The same cold measurement against
invocations with no cache control, and the `Dockerfile` copies the tree before `main` is 5m03s with a 209s `chown`. Filed as #43
running its gates, so on an unchanged tree Docker served those layers from - bust the Docker layer cache for the gate steps, so `script/cibuild`
cache and the build exited 0 having executed nothing. That is not hypothetical and `script/docker` cannot report a green they did not earn
here — every merge this repo has done is a non-fast-forward merge of an (2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts
undiverged branch, so each merge commit's tree is byte-identical to the branch were bare `docker build` invocations with no cache control, and the
head's and each merge CI run was almost certainly a full cache hit; and PR `Dockerfile` copies the tree before running its gates, so on an
#31's reviewer found `make docker` returning success as a 17-layer cache hit, unchanged tree Docker served those layers from cache and the build
catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the exited 0 having executed nothing. That is not hypothetical here —
scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or every merge this repo has done is a non-fast-forward merge of an
break it. `ARG` is scoped per stage and this `Dockerfile` has three gates undiverged branch, so each merge commit's tree is byte-identical to
across two — `make fmt-check` and `make lint` in the lint stage, `make check` the branch head's and each merge CI run was almost certainly a full
in the build stage — so a single declaration would have left one stage cache hit; and PR #31's reviewer found `make docker` returning
silently cacheable; it is declared in both. And BuildKit hashes the expanded success as a 17-layer cache hit, catching it only by being
command, not the declaration, so a declared-but-unreferenced `ARG` invalidates suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing
nothing: each gate `RUN` echoes the epoch, which also puts the value in the `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break
build log as evidence the layer really ran. Placement is below the dependency it. `ARG` is scoped per stage and this `Dockerfile` has three gates
layers on purpose — a build that goes cold every time would be a different across two — `make fmt-check` and `make lint` in the lint stage,
bug, not a fix. Verified by running each script twice back to back on an `make check` in the build stage — so a single declaration would have
unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on left one stage silently cacheable; it is declared in both. And
all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then BuildKit hashes the expanded command, not the declaration, so a
61.1s; `script/docker` 61.1s then 53.4s), and twelve steps were still served declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN`
`CACHED` in the steady state — both `go mod download`s, `apk add`, `adduser`, echoes the epoch, which also puts the value in the build log as
the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy out of evidence the layer really ran. Placement is below the dependency
the lint stage, and the binary copy into the runtime stage. The lint stage layers on purpose — a build that goes cold every time would be a
still gates the build stage: with a deliberate `unused` finding planted in the different bug, not a fix. Verified by running each script twice back
tree, the build failed at `make lint` in 36.1s and the build-stage to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all
`make check` never started. The build stage also still drops to the three gates executed on all four runs, each with a fresh epoch in
unprivileged `builder` user before `make check`, which the suite depends on the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s
rather than merely prefers: forcing the same image to run the tests as root then 53.4s), and twelve steps were still served `CACHED` in the
fails `TestScanHardlinkRunFailsTogether`, because root reads straight through steady state — both `go mod download`s, `apk add`, `adduser`, the
the `chmod(0)` the test uses to prove hard links are read once. This is the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy
local fix only; propagating it to the canonical templates is `prompts` #26 out of the lint stage, and the binary copy into the runtime stage.
- check the installed golangci-lint version in `script/bootstrap` instead of The lint stage still gates the build stage: with a deliberate
only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24): `unused` finding planted in the tree, the build failed at
`missing golangci-lint` meant any linter already on `PATH` satisfied the `make lint` in 36.1s and the build-stage `make check` never started.
check, so the pin was never consulted and the v2.12.2 bump from #3 was inert The build stage also still drops to the unprivileged `builder` user
on every host that already had one — this host ran v2.10.1 against a v2.12.2 before `make check`, which the suite depends on rather than merely
pin, `make check` went green, and `make docker` then rejected the same commit prefers: forcing the same image to run the tests as root fails
with findings the local gate never saw. The version now lives in one place, `TestScanHardlinkRunFailsTogether`, because root reads straight
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a through the `chmod(0)` the test uses to prove hard links are read
bump cannot half-apply; a `golangci_lint_version` helper parses once. This is the local fix only; propagating it to the canonical
`golangci-lint --version` (taking the field after the word `version` and templates is `prompts` #26
tolerating an optional leading `v`, which the module ref carries and the - check the installed golangci-lint version in `script/bootstrap`
binary's output does not), and any version that is not the pin — older, newer, instead of only its presence (2026-08-09, branch
absent or unparseable — is reinstalled. The install is then verified against `bootstrap-version-check`, closes #24): `missing golangci-lint` meant
the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or any linter already on `PATH` satisfied the check, so the pin was never
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on consulted and the v2.12.2 bump from #3 was inert on every host that
`PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or already had one — this host ran v2.10.1 against a v2.12.2 pin,
the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow `make check` went green, and `make docker` then rejected the same
the install and leave the local gate disagreeing with CI under an affirmative commit with findings the local gate never saw. The version now lives
`bootstrap complete`. Bootstrap now re-reads the effective version after in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module
installing and, on a mismatch, prints both paths and both versions to stderr ref derived from it so a bump cannot half-apply; a
and exits non-zero instead of claiming success; it does not reorder anyone's `golangci_lint_version` helper parses `golangci-lint --version`
(taking the field after the word `version` and tolerating an optional
leading `v`, which the module ref carries and the binary's output does
not), and any version that is not the pin — older, newer, absent or
unparseable — is reinstalled. The install is then verified against the
binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
first on `PATH`, so a wrong-version one sitting ahead of it — nix,
apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
stage makes — would swallow the install and leave the local gate
disagreeing with CI under an affirmative `bootstrap complete`.
Bootstrap now re-reads the effective version after installing and, on
a mismatch, prints both paths and both versions to stderr and exits
non-zero instead of claiming success; it does not reorder anyone's
`PATH` or delete their binary. The `--version` call keeps its stderr `PATH` or delete their binary. The `--version` call keeps its stderr
connected, so a present-but-broken binary says why rather than reinstalling connected, so a present-but-broken binary says why rather than
forever in silence, and is bounded by `timeout(1)` where that exists, so a reinstalling forever in silence, and is bounded by `timeout(1)` where
wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their that exists, so a wedged binary cannot hang bootstrap. `git`, `make`
presence-only checks and now say why in a comment: they are host and `go` keep their presence-only checks and now say why in a
package-manager tools the repo deliberately does not pin, with `go.mod` comment: they are host package-manager tools the repo deliberately
governing the language version and the digest-pinned images covering does not pin, with `go.mod` governing the language version and the
reproducible builds. Verified on this host by bootstrapping from v2.10.1 to digest-pinned images covering reproducible builds. Verified on this
v2.12.2 and running it again to a no-op, plus stub runs of the real script host by bootstrapping from v2.10.1 to v2.12.2 and running it again to
under `dash` covering a thirteen-input parse matrix (absent, older, newer, a no-op, plus stub runs of the real script under `dash` covering a
host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit, thirteen-input parse matrix (absent, older, newer, host-style,
impostor binary, `(devel)`, trailing `version`), a shadowed install that must image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor
exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a binary, `(devel)`, trailing `version`), a shadowed install that must
wedged binary that must hit the timeout; `make check` and `make lint` are exit non-zero, an install destination not on `PATH` at all, `GOBIN`
clean at v2.12.2, so v2.10.1 was not hiding any findings on `main` set, and a wedged binary that must hit the timeout; `make check` and
`make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch - unwind the hash worker pool on the error path (2026-08-09, branch
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment `hash-pool-cleanup`, closes #6): `hashPhase` used to return the
`recordRun` failed and abandon the pool — the feeder parked forever on a full moment `recordRun` failed and abandon the pool — the feeder parked
`jobs` channel and every worker on a full `results` channel. That only stopped forever on a full `jobs` channel and every worker on a full
being invisible when #4 landed and `runScan` began unwinding instead of `results` channel. That only stopped being invisible when #4 landed
calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every and `runScan` began unwinding instead of calling `os.Exit`. The
blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is pool is now an owned, context-aware `hashPool`: every blocking send
closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels in the feeder and the workers selects on `ctx.Done()`, `jobs` is
and then drains `results` until the last goroutine has exited — draining is closed on every path out, and `hashPhase` defers `pool.stop()`,
what frees a worker already parked on a send. `ctx` is threaded from which cancels and then drains `results` until the last goroutine
`cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole has exited — draining is what frees a worker already parked on a
database layer (it is the first parameter everywhere), so #5 can hand this send. `ctx` is threaded from `cmd.Context()` through `runScan`,
path a signal and needs to add nothing else. The walk pool never leaked, `syncScan`, both worker pools and the whole database layer (it is
because `walkPhase` always drains its events to close, but it has the same the first parameter everywhere), so #5 can hand this path a signal
unbounded-send shape and #5 will give it an early return, so it gets the same and needs to add nothing else. The walk pool never leaked, because
treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a `walkPhase` always drains its events to close, but it has the same
partial size census, and every file it never reached looks vanished to the unbounded-send shape and #5 will give it an early return, so it
update phase. That phase's own `BeginTx` fails on the same cancelled context gets the same treatment plus a `ctx.Err()` guard after the walk: a
before deleting anything, so the guard is defence in depth rather than the cancelled walk yields a partial size census, and every file it never
only barrier — but it is the one that survives #5 deciding an interrupted scan reached looks vanished to the update phase. That phase's own
may commit what it has. Tests drive `run(scan)` against a database whose `BeginTx` fails on the same cancelled context before deleting
insert trigger aborts, and assert both that the scan fails instead of hanging anything, so the guard is defence in depth rather than the only
and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a barrier — but it is the one that survives #5 deciding an interrupted
second set cancels a scan part-way through the walk — deterministically, by scan may commit what it has. Tests drive `run(scan)` against a
counting the scan's own consultations of `ctx.Done()` rather than racing a database whose insert trigger aborts, and assert both that the scan
timer — and asserts that it stops at the guard holding a partial census and a fails instead of hanging and that `runtime.NumGoroutine()` polls
still-populated record index, with every record intact. The remaining back to its pre-scan baseline; a second set cancels a scan part-way
cancellation branches of both pools are covered by direct tests of through the walk — deterministically, by counting the scan's own
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker` consultations of `ctx.Done()` rather than racing a timer — and
and `hashPhase` asserts that it stops at the guard holding a partial census and a
- guarantee the database is closed on every fatal exit path (2026-08-09, branch still-populated record index, with every record intact. The
`db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so remaining cancellation branches of both pools are covered by direct
the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now tests of `sendEvent`, the walk workers, `dispatchDirs`,
actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`, `feedHashJobs`, `hashWorker` and `hashPhase`
`loadRecords` and `resolveRoots` return errors instead. The single exit point - guarantee the database is closed on every fatal exit path
is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned) (2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
runtime failure from being reported as a usage error or printing the usage it the SQLite WAL checkpoint — now actually runs when a subcommand
text. New `main_test.go` drives the CLI in-process and asserts the exit codes fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
from README §Error handling plus the stdout/stderr split, including that a `resolveRoots` return errors instead. The single exit point is `run`
fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar in `main.go`: it maps a `fatalError` (anything a subcommand
behind for `scan`, `report` or `trees` returned) to exit 1 and cobra's own argument and flag errors to exit
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch 2, which keeps a runtime failure from being reported as a usage
`golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter error or printing the usage text. New `main_test.go` drives the CLI
in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, in-process and asserts the exit codes from README §Error handling
and replaced `.golangci.yml` with the canonical file — the linter settings plus the stdout/stderr split, including that a fatal error raised
after the database is open leaves no `-wal`/`-shm` sidecar behind
for `scan`, `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
closes #3): bumped the pinned linter in the `Dockerfile` lint
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
`.golangci.yml` with the canonical file — the linter settings
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under (`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
`linters.settings` per the v2 schema, so they are actually applied; no new `linters.settings` per the v2 schema, so they are actually
lint findings surfaced applied; no new lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints - convert Makefile targets to scripts-to-rule-them-all `script/`
like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12 entrypoints like the other managed repos (2026-07-26, commit
`script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`, `3abeacf`, closes #1): all 12 `script/` entrypoints exist
`lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
`install-precommit`) and every Makefile target is now a thin shim over them, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
matching the other managed repos `install-precommit`) and every Makefile target is now a thin shim
over them, matching the other managed repos
- make the binary the default Make target (2026-07-24, branch - make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran `make-default-target`): plain `make` now builds `sfdupes`
`check` plus `build`); `make build` remains as an alias (previously it ran `check` plus `build`); `make build` remains as
- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch an alias
`scan-wide-phases`): all operands seed the shared walk pool and every pass - scan-wide phases, concurrent operands, batched updates (2026-07-24,
runs once over the whole scan, so totals and ETAs are scan-global; the branch `scan-wide-phases`): all operands seed the shared walk pool
per-operand walk/hash/update cycles and their stderr announcements are gone; and every pass runs once over the whole scan, so totals and ETAs
the update pass commits in batched transactions — the filesystem is are scan-global; the per-operand walk/hash/update cycles and their
authoritative and the database an eventually-consistent reflection, so stderr announcements are gone; the update pass commits in batched
scan-level atomicity is not required transactions — the filesystem is authoritative and the database an
eventually-consistent reflection, so scan-level atomicity is not
required
- split the stat pass back out of the walk (2026-07-24, branch - split the stat pass back out of the walk (2026-07-24, branch
`parallel-phases`): phases are strictly sequential again — walk, stat, hash, `parallel-phases`): phases are strictly sequential again — walk,
update per operand — with parallelism only inside each phase; the walk stat, hash, update per operand — with parallelism only inside each
enumerates paths with per-directory workers and the stat pass lstats them with phase; the walk enumerates paths with per-directory workers and the
per-file workers, restoring the exact total/ETA stat bar stat pass lstats them with per-file workers, restoring the exact
- announce each operand on stderr before its passes (2026-07-24, branch total/ETA stat bar
`scan-operand-progress`): with per-operand walk/hash/update cycles, a - announce each operand on stderr before its passes (2026-07-24,
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the branch `scan-operand-progress`): with per-operand walk/hash/update
whole run's — an operator watching operand 3 of 14 hash 300k files concluded cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
20M files were being skipped that looked like the whole run's — an operator watching operand 3 of
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single 14 hash 300k files concluded 20M files were being skipped
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in - parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
4h on a ZFS server); it is now a per-directory worker-pool traversal that was a single goroutine and took hours at ~20M files on a busy pool
records size/mtime during the walk (folding away the separate stat pass, (observed: 22M files in 4h on a ZFS server); it is now a
halving metadata I/O), and each `PATH` operand commits in its own transaction per-directory worker-pool traversal that records size/mtime during
so an interrupted scan keeps completed operands the walk (folding away the separate stat pass, halving metadata
I/O), and each `PATH` operand commits in its own transaction so an
interrupted scan keeps completed operands
- persistent scan database (2026-07-24, branch `persistent-database`): `scan` - persistent scan database (2026-07-24, branch `persistent-database`):
now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays `scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
disabled) keyed by absolute path that survives between runs — a rescan hashes Go, cgo stays disabled) keyed by absolute path that survives between
only new or changed files (by mtime/size), deletes records for files vanished runs — a rescan hashes only new or changed files (by mtime/size),
from under the scanned operands, and leaves records outside them untouched, so deletes records for files vanished from under the scanned operands,
`scan` can be cronned daily; `report` and `trees` read the database (no and leaves records outside them untouched, so `scan` can be cronned
positional arguments) instead of a scan stream. Database at daily; `report` and `trees` read the database (no positional
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL arguments) instead of a scan stream. Database at
journaling plus a single-transaction update keep a report run during a scan `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
safe WAL journaling plus a single-transaction update keep a report run
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`, during a scan safe
and push `main` plus tags (2026-07-23) - add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
`v0.0.1`, and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required - `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
`PATH...` operands via cobra flags replacing the `/srv` `-root` default; new `PATH...` operands via cobra flags replacing the `/srv` `-root`
`-x`/`--one-file-system` flag (GNU convention) to stop at filesystem default; new `-x`/`--one-file-system` flag (GNU convention) to stop
boundaries, which are crossed by default at filesystem boundaries, which are crossed by default
- bring the repo into full policy compliance (2026-07-23, branch - bring the repo into full policy compliance (2026-07-23, branch
`repo-policy-compliance`; checklist below) `repo-policy-compliance`; checklist below)
- `git init` with README-only first commit; code baseline committed on `main` - `git init` with README-only first commit; code baseline committed on
(2026-07-22) `main` (2026-07-22)
- implement `scan`, `report`, and `trees` subcommands (pre-git history) - implement `scan`, `report`, and `trees` subcommands (pre-git history)
# Future Steps # Future Steps
- possible later features (explicitly out of scope per README): full-content - possible later features (explicitly out of scope per README):
verification of candidates, removal-script helpers full-content verification of candidates, removal-script helpers
# Repo Policy Compliance # Repo Policy Compliance
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
formatting commit is needed. standalone formatting commit is needed.
- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit - [x] `.gitignore` missing — the compiled `sfdupes` binary and
untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns `files.dat` sit untracked in the tree; needs OS/editor/Go
artifacts plus secrets patterns
- [x] `.editorconfig` missing - [x] `.editorconfig` missing
- [x] `LICENSE` missing and README has no License section (MIT assumed from - [x] `LICENSE` missing and README has no License section (MIT assumed
house convention — user to confirm) from house convention — user to confirm)
- [x] `REPO_POLICIES.md` missing from repo root - [x] `REPO_POLICIES.md` missing from repo root
- [x] `.golangci.yml` missing (install canonical copy); code must then pass - [x] `.golangci.yml` missing (install canonical copy); code must then
`make lint` (150 findings fixed; `make lint` is clean) pass `make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`, - [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
`docker`, `hooks`; `check` currently depends on `build`, which writes the `fmt-check`, `docker`, `hooks`; `check` currently depends on
binary (`make check` must not modify files) `build`, which writes the binary (`make check` must not modify
- [x] no tests — `go test ./...` has nothing to run; policy requires real tests files)
with a 30-second timeout and the conditional `-v` rerun pattern (suite - [x] no tests — `go test ./...` has nothing to run; policy requires
covers parsing, grouping, digests, suppression, hashing, and the scan real tests with a 30-second timeout and the conditional `-v`
pipeline; 64% coverage) rerun pattern (suite covers parsing, grouping, digests,
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast suppression, hashing, and the scan pipeline; 64% coverage)
lint stage, build stage running `make check` - [x] `Dockerfile` missing — Go multistage with hash-pinned images:
fail-fast lint stage, build stage running `make check`
- [x] `.dockerignore` missing - [x] `.dockerignore` missing
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout - [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
action pinned by commit SHA) checkout action pinned by commit SHA)
- [x] README lacks required sections: Description first line - [x] README lacks required sections: Description first line
(name/purpose/category/license/author), Getting Started, Rationale, TODO, (name/purpose/category/license/author), Getting Started,
License, Author Rationale, TODO, License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now that the - [x] README non-goal "no git repository setup and no CI" is stale now
repo is under git with CI that the repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target exists) - [x] pre-commit hook not installed (`make hooks` once the target
exists)
Accepted divergences (no action): Accepted divergences (no action):
- flat single-package layout with `.go` files in the repo root — fine for a - flat single-package layout with `.go` files in the repo root — fine
small single-binary tool per the Go styleguide; the tracker audit agrees for a small single-binary tool per the Go styleguide; the tracker
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go audit agrees
builds) and the race detector requires cgo - `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
(pure-Go builds) and the race detector requires cgo
+23 -2
View File
@@ -174,9 +174,30 @@ func initSchema(ctx context.Context, db *sql.DB) error {
} }
// createSchema applies the schema to a fresh database and stamps the // createSchema applies the schema to a fresh database and stamps the
// schema version. // schema version. A database with user_version 0 that already has a
// files table was not created by this build — a foreign or partially
// initialized file. Adopting it silently could corrupt unrelated data,
// so that is a fatal schema-version error telling the operator to
// remove the file and rescan.
func createSchema(ctx context.Context, db *sql.DB) error { func createSchema(ctx context.Context, db *sql.DB) error {
_, err := db.ExecContext(ctx, createTableSQL) var name string
err := db.QueryRowContext(ctx,
"SELECT name FROM sqlite_master "+
"WHERE type = 'table' AND name = 'files'").Scan(&name)
switch {
case err == nil:
return fmt.Errorf(
"has a files table but no schema version; "+
"remove the file and rescan: %w", errSchemaVersion)
case errors.Is(err, sql.ErrNoRows):
// Genuinely empty: create the schema below.
default:
return fmt.Errorf("check for files table: %w", err)
}
_, err = db.ExecContext(ctx, createTableSQL)
if err != nil { if err != nil {
return fmt.Errorf("create schema: %w", err) return fmt.Errorf("create schema: %w", err)
} }
+32
View File
@@ -78,6 +78,38 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
} }
} }
func TestOpenScanDatabaseUnversionedForeign(t *testing.T) {
t.Parallel()
path := testDBPath(t)
// A database that has a files table but user_version 0 — a foreign
// or partially initialized file. scan must refuse it with a clear
// schema-version error, not adopt it and not emit a raw SQLite
// "table files already exists".
db, err := openDB(path)
if err != nil {
t.Fatal(err)
}
_, err = db.ExecContext(t.Context(), "CREATE TABLE files (x INTEGER)")
if err != nil {
t.Fatal(err)
}
_ = db.Close()
_, err = openScanDatabase(t.Context(), path)
if !errors.Is(err, errSchemaVersion) {
t.Fatalf("err = %v, want errSchemaVersion", err)
}
if !strings.Contains(err.Error(), "remove the file and rescan") {
t.Fatalf("err = %v, want it to tell the operator to remove and rescan",
err)
}
}
func TestOpenReportDatabaseMissing(t *testing.T) { func TestOpenReportDatabaseMissing(t *testing.T) {
t.Parallel() t.Parallel()
-5
View File
@@ -1,5 +0,0 @@
{
"devDependencies": {
"prettier": "3.8.1"
}
}
-31
View File
@@ -11,12 +11,6 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# yarn provides prettier, which formats Markdown. yarn is a tool, like
# node/git/make/go below; the reference that governs formatting output is
# prettier, pinned by yarn.lock's integrity hash and installed by
# `yarn install --frozen-lockfile`.
YARN_VERSION="1.22.22"
PKGMGR="" PKGMGR=""
SUDO="" SUDO=""
APT_UPDATED="" APT_UPDATED=""
@@ -64,21 +58,6 @@ missing() {
! command -v "$1" >/dev/null 2>&1 ! command -v "$1" >/dev/null 2>&1
} }
ensure_node() {
if ! missing node; then return 0; fi
pkg_install nodejs nodejs node nodejs
}
ensure_yarn() {
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable >/dev/null 2>&1 || true
corepack prepare "yarn@$YARN_VERSION" --activate
else
pkg_install yarn yarn yarn yarn
fi
}
main() { main() {
cd "$ROOT" cd "$ROOT"
@@ -92,16 +71,6 @@ main() {
if missing make; then pkg_install gnumake make make make; fi if missing make; then pkg_install gnumake make make make; fi
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
# node runs prettier and is an unpinned host tool for the same reason
# git/make/go are: it comes from the host package manager, whatever
# version it ships. It is not installed via nvm the way the canonical
# template does, because nvm's prebuilt node is glibc-linked and does
# not run on this repo's musl/Alpine build image. prettier — the tool
# whose version affects formatting output — is pinned by yarn.lock.
ensure_node
ensure_yarn
yarn install --frozen-lockfile
# Linting runs via docker only (script/lint), so docker is a lint # Linting runs via docker only (script/lint), so docker is a lint
# prerequisite rather than something bootstrap installs. Warn, do # prerequisite rather than something bootstrap installs. Warn, do
# not fail: everything except `make lint` — and, through it, # not fail: everything except `make lint` — and, through it,
+1 -12
View File
@@ -1,23 +1,12 @@
#!/bin/sh #!/bin/sh
# script/fmt: format all files (writes). gofmt for Go, prettier for # script/fmt: format all files (writes).
# Markdown. prettier is the pinned devDependency in package.json/
# yarn.lock; script/bootstrap installs it (see run_prettier).
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
run_prettier() {
if ! command -v yarn >/dev/null 2>&1; then
echo "fmt: yarn not found; run script/bootstrap first" >&2
exit 1
fi
yarn run prettier "$@"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
gofmt -s -w . gofmt -s -w .
run_prettier --write '**/*.md' --tab-width 4 --prose-wrap always
} }
main "$@" main "$@"
+2 -21
View File
@@ -1,37 +1,18 @@
#!/bin/sh #!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as # script/fmt-check: check formatting (read-only). Same scope as
# script/fmt: gofmt for Go, prettier for Markdown. Both run every time # script/fmt, but fails instead of writing.
# and each reports independently, so a failure names which formatter is
# unhappy; the script exits non-zero if either found unformatted files.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
run_prettier() {
if ! command -v yarn >/dev/null 2>&1; then
echo "fmt-check: yarn not found; run script/bootstrap first" >&2
exit 1
fi
yarn run prettier "$@"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
rc=0
files="$(gofmt -s -l .)" files="$(gofmt -s -l .)"
if [ -n "$files" ]; then if [ -n "$files" ]; then
echo "gofmt: files not formatted:" >&2 echo "gofmt: files not formatted:" >&2
echo "$files" >&2 echo "$files" >&2
rc=1 exit 1
fi fi
if ! run_prettier --check '**/*.md' --tab-width 4 --prose-wrap always; then
echo "prettier: Markdown not formatted; run make fmt" >&2
rc=1
fi
exit "$rc"
} }
main "$@" main "$@"
-8
View File
@@ -1,8 +0,0 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
prettier@3.8.1:
version "3.8.1"
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==