Compare commits

4 Commits
Author SHA1 Message Date
sneak 32d8252e50 Format Markdown with prettier in make fmt and make fmt-check (closes #19)
check / check (push) Failing after 1s
script/fmt and script/fmt-check run prettier over every Markdown file
again, next to gofmt. prettier is pinned by hash through package.json
and yarn.lock, copied from the prompts repo with .prettierrc and
.prettierignore, and is never installed on a host: a new prettier stage
of the Dockerfile installs it into a digest-pinned node image, and both
scripts build that stage and run it with the repository mounted. CI
checks the Markdown in a markdown stage that the build stage waits on.
Because make fmt-check now runs docker, the Dockerfile runs gofmt
directly in its lint stage instead. All Markdown is reformatted.

Model: opus-5-5
2026-10-04 14:49:17 +00:00
clawbot e2227ac07e Copy the current canonical .golangci.yml (closes #26)
check / check (push) Failing after 10s
The shared lint config in the prompts repo moved from the deprecated
gomodguard linter to gomodguard_v2, with a module block list, and now
enables depguard to keep test-support packages out of non-test files.
This replaces the repo's copy with that file unchanged, so lint no
longer prints the gomodguard deprecation warning.

Model: opus-5-5
2026-10-04 16:13:20 +02:00
clawbot 4a16a41bd7 Test both hashWorker cancellation checks on their own (closes #83)
check / check (push) Failing after 2s
TestHashWorkerDropsQueuedRuns now passes hashWorker a hash function
that records being called, so a worker that hashes a run after the
scan is cancelled fails the test every time instead of only when it
then chose to send its result.

TestHashWorkerAbandonsBlockedSend cancels the scan from inside the
hash function and leaves the result channel unread, so the worker can
only return through the cancellation case beside its send. The scan
tests could not show this, because stop drains results and frees a
parked worker anyway.

Model: opus-5-5
2026-10-04 16:01:36 +02:00
clawbot 722675f153 Escape the database path in the SQLite connection string (closes #55)
check / check (push) Failing after 2s
openDB put the path into the connection string unescaped, so a ? or #
in it ended the file name and a % started an escape: scan could
silently fill a database under a shortened name. The path now goes
through net/url as a file: URI. An absolute path gets an empty host and
a relative path none, because SQLite reads what follows file:// up to
the next slash as a host name. The path is not cleaned, so it stays
exactly what the operator gave.

A test runs scan, report and trees against such a file name given as an
absolute path, as one starting with //, and as a relative path, and
checks that only that file and its lock file exist afterwards.

Model: opus-5-5
2026-10-04 15:13:19 +02:00
7 changed files with 219 additions and 27 deletions
+66 -2
View File
@@ -10,14 +10,20 @@ run:
linters:
default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable:
# Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields
- depguard # Dependency allow/block lists
- godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings:
lll:
line-length: 88
@@ -28,6 +34,64 @@ linters:
max-complexity: 15
dupl:
threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues:
max-issues-per-linter: 0
+3 -1
View File
@@ -263,7 +263,9 @@ All three subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
command-line flag.
command-line flag. The path names the file exactly, whatever characters it
holds (`?`, `#` and `%` included); a relative path is relative to the working
directory.
- `scan` creates the database (and its parent directory) on first use. `report`
and `trees` require an existing database; a missing database file is a fatal
error (exit 1) telling the user to run `scan` first.
+10
View File
@@ -32,6 +32,16 @@
CI checks it; all Markdown reformatted (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/19)
- `.golangci.yml` replaced with the current canonical copy, which uses
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/26)
- a test fails when either `hashWorker` cancellation check in `scan.go` is
removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)
- a database path holding `?`, `#` or `%` opens exactly the file it names
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
- `scan` rejects `--workers` below 1 as a usage error instead of running
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
+52 -22
View File
@@ -729,47 +729,77 @@ func TestFeedHashJobsClosesJobsWhenCancelled(t *testing.T) {
// TestHashWorkerDropsQueuedRuns checks that a cancelled hash worker
// keeps reading jobs and drops the runs rather than reading files
// nobody wants the hashes of — while still letting the range run out
// so the pool tears down. The queued run names a file that does not
// exist, so a worker that hashed it anyway would produce a result.
//
// hashWorker's other cancellation exit, abandoning the send of a
// result, is reachable from the scan: stop cancels the pool before it
// drains results, so a worker waiting on that send can leave through
// it. The tests that stop a scan mid-hash, among them
// TestScanHashWriteFailureUnwindsPool, reach it in some runs only,
// depending on timing, and no test fails without it, since stop's
// drain frees a waiting worker anyway. This test, for its part, catches
// a removed drop check in some runs only: a worker that hashes the run
// anyway then picks at random between sending the result and leaving.
// so the pool tears down. The hash function records that it was
// called, so a worker that hashed the queued run anyway is caught
// every time.
func TestHashWorkerDropsQueuedRuns(t *testing.T) {
t.Parallel()
done := make(chan struct{})
jobs := make(chan []fileRec, 1)
results := make(chan hashResult, 1)
results := make(chan hashResult)
run := []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
jobs <- run
jobs <- []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
close(jobs)
var hashed atomic.Bool
hash := func(path string, size int64) (string, string, string, error) {
hashed.Store(true)
return hashSignature(path, size)
}
go func() {
defer close(done)
hashWorker(cancelledContext(t), jobs, results, hashSignature)
hashWorker(cancelledContext(t), jobs, results, hash)
}()
awaitReturn(t, done, "hashWorker")
select {
case r := <-results:
t.Errorf("cancelled hash worker produced %+v, want the run dropped",
r)
default:
if hashed.Load() {
t.Error("cancelled hash worker hashed the queued run, want it dropped")
}
}
// TestHashWorkerAbandonsBlockedSend checks that a hash worker with a
// result to deliver and nobody to deliver it to leaves once the scan
// is cancelled, instead of holding the pool open. The scan tests do
// not catch this: stop drains results, which frees a parked worker
// anyway.
func TestHashWorkerAbandonsBlockedSend(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan struct{})
jobs := make(chan []fileRec, 1)
// Unbuffered and unread, with jobs left open: the worker's only way
// out is the cancellation case beside its send.
results := make(chan hashResult)
jobs <- []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
// The scan is cancelled while the worker hashes, so the worker has
// already passed the check that drops queued runs.
hash := func(path string, size int64) (string, string, string, error) {
cancel()
return hashSignature(path, size)
}
go func() {
defer close(done)
hashWorker(ctx, jobs, results, hash)
}()
awaitReturn(t, done, "hashWorker")
}
// TestHashPhaseCancelledReturnsContextError checks the result loop's
// own exit: with the pool cancelled, no result will ever arrive, and
// the loop must leave through the cancellation rather than wait for a
+14 -1
View File
@@ -6,6 +6,7 @@ import (
"errors"
"fmt"
"io/fs"
"net/url"
"os"
"path/filepath"
"slices"
@@ -107,7 +108,19 @@ const reportParams = "mode=ro" +
// openDB opens the SQLite database at path with the connection
// parameters params. It does not create or verify the schema.
func openDB(path, params string) (*sql.DB, error) {
db, err := sql.Open("sqlite", "file:"+path+"?"+params)
// The path is escaped into a file: URI, so ?, # and % in it stay
// part of the file name. SQLite reads what follows file:// up to
// the next / as a host name, so an absolute path goes after an
// empty host (file:///abs) and a relative path goes without one
// (file:rel).
uri := url.URL{
Scheme: "file",
OmitHost: !filepath.IsAbs(path),
Path: path,
RawQuery: params,
}
db, err := sql.Open("sqlite", uri.String())
if err != nil {
return nil, fmt.Errorf("open database %s: %w", path, err)
}
+68
View File
@@ -8,6 +8,7 @@ import (
"io/fs"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
@@ -639,6 +640,73 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
}
}
// assertRunsUseDatabase runs scan, then report and trees, against the
// database that SFDUPES_DATABASE names, the file name in dir. It fails
// unless the reports find the duplicate pair the scan recorded and dir
// then holds only that file and its lock file: nothing was created
// under a shortened name.
func assertRunsUseDatabase(t *testing.T, dir, name string) {
t.Helper()
dupes := scanFixture(t)
want := "first\tdupe\tsize\n" + dupes[0] + "\t" + dupes[1] + "\t300\n"
if got := runStdout(t, cmdReport); got != want {
t.Errorf("report stdout = %q, want %q", got, want)
}
want = "first\tdupe\tfiles\tsize\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
if got := runStdout(t, cmdTrees); got != want {
t.Errorf("trees stdout = %q, want %q", got, want)
}
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
got := make([]string, 0, len(entries))
for _, e := range entries {
got = append(got, e.Name())
}
if wantFiles := []string{name, name + ".lock"}; !slices.Equal(got, wantFiles) {
t.Errorf("%s holds %q, want %q", dir, got, wantFiles)
}
}
func TestRunDatabasePathUsedAsGiven(t *testing.T) {
// README §Database: the path names the database file exactly. In
// SQLite's connection string an unescaped ? or # would end the file
// name and % would start an escape, and a path starting with //
// could be read as a host name. %25 is a valid escape, so unescaped
// this name opens a file named a without any error.
const name = "a?b#c%25d e.sqlite"
t.Run("absolute", func(t *testing.T) {
dir := t.TempDir()
t.Setenv(databaseEnv, filepath.Join(dir, name))
assertRunsUseDatabase(t, dir, name)
})
t.Run("leading double slash", func(t *testing.T) {
dir := t.TempDir()
t.Setenv(databaseEnv, "/"+filepath.Join(dir, name))
assertRunsUseDatabase(t, dir, name)
})
t.Run("relative", func(t *testing.T) {
dir := t.TempDir()
t.Chdir(dir)
t.Setenv(databaseEnv, name)
assertRunsUseDatabase(t, dir, name)
})
}
// holdScanLock takes the lock on the database at path, as a running
// scan does, and holds it until the test ends. It fails the test when
// the lock is already held.
+6 -1
View File
@@ -12,7 +12,12 @@ main() {
cd "$ROOT"
status=0
files="$(gofmt -s -l .)"
# Under set -e a bare assignment would end the script when gofmt
# fails (a Go file it cannot parse), and prettier would never run.
if ! files="$(gofmt -s -l .)"; then
echo "gofmt: failed; see its errors above" >&2
status=1
fi
if [ -n "$files" ]; then
echo "gofmt: files not formatted:" >&2
echo "$files" >&2