Author SHA1 Message Date
sneak 207bd918c2 Rewrite TODO.md sentence by sentence (closes #135)
check / check (push) Canceled after 0s
Rewrites `TODO.md` from scratch for
#135, by the rules applied to
`README.md` in #140. It is the
project's own task list, so every completed step keeps its date, its
issue or branch and every fact that the old entry recorded, in short
sentences, with lists as bullets. An entry superseded later says so in
one sentence.

`TODO.md` records history, so its entries were not corrected against the
code. The 2026-10-17 release-rule entry is known to be stale. One product
name in the 2026-10-15 entry is replaced by a description.

This file carries the `TODO.md` entry of
#140 word for word, so the
conflict once that merges is resolved by taking this file whole.

Model: fable-5-1
2026-10-26 17:14:55 +00:00
clawbot ee09cda355 Never scan an operand inside a .zfs directory (closes #142)
check / check (push) Canceled after 0s
An operand inside a directory named .zfs was walked, because the
operand check looked only at the operand's own name. The check now
also looks at every directory above the operand, both as given and
with its symlinks resolved. Such an operand gets a warning, counts as
skipped and is not walked, whether given alone or under another
operand. The check runs before overlapping operands are pruned, so an
operand under another one still gets its warning.

The test for an operand named .zfs now writes its record beneath .zfs
straight into the database, since no scan stores one there any more.

Judgement call: a path component named .zfs counts whatever it is, a
symlink included.

Model: opus-5-5
2026-10-26 15:15:37 +01:00
4 changed files with 276 additions and 125 deletions
+20 -11
View File
@@ -461,17 +461,26 @@ exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
`-x`, is on a different filesystem than that operand, because that operand's `-x`, is on a different filesystem than that operand, because that operand's
walk does not enter such a directory and so does not reach it. walk does not enter such a directory and so does not reach it.
An operand that is a symlink (never followed, not even as an operand), socket, An operand is not scanned when it is:
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
a one-line warning naming the path and what it is, counts it as skipped, and - a symlink, which `scan` never follows, not even as an operand;
drops it from the scanned operands before reading the database. Another operand - a socket, FIFO or device node;
beneath it is still scanned. The records stored beneath it are not deleted: they - at or inside a directory named `.zfs`.
are treated like any other record outside the scanned operands. If it lies under
another operand, they are under that operand instead, and are treated like any An operand reached through a symlink into a `.zfs` directory is not scanned
other record there. This is not an error: a scan whose every operand is dropped either.
walks nothing and exits 0. An operand that is not on a persistent filesystem
(see "Rules for the walk") is dropped the same way. The line `scan` prints for This holds even when the operand lies under another operand. `scan` prints a
it is not a warning, though, and it is not counted as skipped. one-line warning naming the path and what it is, counts it as skipped, and drops
it from the scanned operands before reading the database. An operand beneath a
symlink operand is still scanned. The records stored beneath the dropped operand
are not deleted: they are treated like any other record outside the scanned
operands. If it lies under another operand, they are under that operand instead,
and are treated like any other record there. This is not an error: a scan whose
every operand is dropped walks nothing and exits 0. An operand that is not on a
persistent filesystem (see "Rules for the walk") is dropped the same way. The
line `scan` prints for it is not a warning, though, and it is not counted as
skipped.
`scan` synchronizes the database with the filesystem state under the scanned `scan` synchronizes the database with the filesystem state under the scanned
operands: operands:
+106 -92
View File
@@ -38,8 +38,14 @@ are known.
it. It leads with what the program is and how to use it. A reference section it. It leads with what the program is and how to use it. A reference section
covers every setting and every subcommand. Development and testing come last. covers every setting and every subcommand. Development and testing come last.
Every statement was checked against the code on `next`. Five comments in Every statement was checked against the code on `next`. Five comments in
`main_test.go` cite "Settings", where their rules now live (2026-10-26, `main_test.go` cite "Settings", where their rules now live. One comment each
https://git.eeqj.de/sneak/sfdupes/issues/135) in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README
now says (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
- an operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. `scan` warns about it, counts it as
skipped and does not walk it (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/142)
- A file or directory removed while the scan runs is not an error, in the walk - A file or directory removed while the scan runs is not an error, in the walk
or in the hash phase. The scan prints no line for it and does not count it as or in the hash phase. The scan prints no line for it and does not count it as
@@ -57,10 +63,10 @@ are known.
`trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26, `trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/134) https://git.eeqj.de/sneak/sfdupes/issues/134)
- Every file the walk records gets three hashes in the hash phase. `head` and - The hash phase gives three hashes to every file that the walk records. `head`
`tail` are the SHA-256 of its first and last 64 KiB. `content` is the SHA-256 and `tail` are the SHA-256 of its first and last 64 KiB. `content` is the
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB SHA-256 of the whole file under 100 MiB, and of 1 MiB at each gigabyte from
up. Gone are: 100 MiB up. Gone are:
- the 10 MiB and 50 MiB thresholds; - the 10 MiB and 50 MiB thresholds;
- the size census, which counted the sizes of every walked file and of every - the size census, which counted the sizes of every walked file and of every
record outside the operands. A file was hashed only when the census showed record outside the operands. A file was hashed only when the census showed
@@ -73,8 +79,8 @@ are known.
with a content hash, since every record has all three (2026-10-25, with a content hash, since every record has all three (2026-10-25,
https://git.eeqj.de/sneak/sfdupes/issues/132) https://git.eeqj.de/sneak/sfdupes/issues/132)
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` copied again - `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` were copied
from `sneak/prompts` `next` at `0653579`. Two steps now come before again from `sneak/prompts` `next` at `0653579`. Two steps now come before
`ARG CACHEBUST`: `ARG CACHEBUST`:
- the module download in the `lint`, `test` and `builder` stages of the - the module download in the `lint`, `test` and `builder` stages of the
`Dockerfile`; `Dockerfile`;
@@ -150,12 +156,12 @@ are known.
the default excludes only, and `scan --help` lists them (2026-10-17, the default excludes only, and `scan --help` lists them (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/124) https://git.eeqj.de/sneak/sfdupes/issues/124)
- The rate a progress display shows covers the items completed over the last 5 - A progress display's rate covers the items completed over the last 5 seconds.
seconds. When stderr is not a terminal, it covers the items since the line When stderr is not a terminal, it covers the items since the line before. It
before. It falls to 0 while nothing completes. A burst after a wait no longer falls to 0 while nothing completes. A burst after a wait no longer reads as a
reads as a steady rate of over 1,000 files a second. The time remaining still steady rate of over 1,000 files a second. The time remaining still comes from
comes from the phase's average. Once no item has completed for 10 seconds, the the phase's average. Once no item has completed for 10 seconds, the line shows
word `stalled` and where the phase waits take the rate's place, as `stalled` and where the phase waits in place of the rate, as
`stalled disk/db`. The places are: `stalled disk/db`. The places are:
- `disk`; - `disk`;
- `db`; - `db`;
@@ -170,12 +176,12 @@ are known.
- the elapsed time; - the elapsed time;
- the ETA, the time remaining. - the ETA, the time remaining.
On an 80-column terminal, at counts like those of README's example, the The content phase had one display for its checks and one for its reads. Both
content phase's displays for its checks and its reads lost those words. That lost those words on an 80-column terminal, at counts like those of README's
happened for most elapsed times and ETAs in minutes. Every other phase kept example. That happened for most elapsed times and ETAs in minutes. Every
them whenever the elapsed time and the ETA were both in minutes. The content other phase kept them whenever the elapsed time and the ETA were both in
phase writes its last partial batch, the records not yet written, under a minutes. The content phase writes its last partial batch, the records not
display of its own (2026-10-16, yet written, under a display of its own (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/125) https://git.eeqj.de/sneak/sfdupes/issues/125)
- New subcommand `info` prints the database's details on stdout, one per line as - New subcommand `info` prints the database's details on stdout, one per line as
@@ -213,14 +219,20 @@ are known.
opening the database, also while it waits on a busy one. `read` counts the opening the database, also while it waits on a busy one. `read` counts the
records, then reads them against that count. `group` counts the directories it records, then reads them against that count. `group` counts the directories it
groups, and `write` the groups it writes. A display's rate names what its groups, and `write` the groups it writes. A display's rate names what its
phase counts, such as records or groups. stdout and the summary line are phase counts:
unchanged. When stdout and stderr are the same file, as one terminal is, no - files;
line of stdout shares a line with the display. When they are not, the display - records;
keeps being drawn while stdout waits on a slow reader (2026-10-16, - directories;
https://git.eeqj.de/sneak/sfdupes/issues/113) - groups.
stdout and the summary line are unchanged. When stdout and stderr are the
same file, as one terminal is, no line of stdout shares a line with the
display. When they are not, the display keeps being drawn while stdout waits
on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113)
- Each `scan` phase's progress display names the phase and its place among the - Each `scan` phase's progress display names the phase and its place among the
five, as `[2/5] walk`. It runs from the moment the phase starts and shows: five, as `[2/5] walk`. It runs from the moment that the phase starts. It
shows:
- the progress; - the progress;
- the elapsed time; - the elapsed time;
- the ETA, the time remaining. - the ETA, the time remaining.
@@ -245,8 +257,8 @@ are known.
"Database" says when `scan` writes each file (2026-10-16, "Database" says when `scan` writes each file (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/111) https://git.eeqj.de/sneak/sfdupes/issues/111)
- The canonical files and model scripts copied again from `sneak/prompts` `next` - The canonical files and model scripts were copied again from `sneak/prompts`
at `e3f79c4`: `next` at `e3f79c4`:
- `golangci-lint` v2.14.0; - `golangci-lint` v2.14.0;
- lint and test are phases of the `Dockerfile` that write no image, and - lint and test are phases of the `Dockerfile` that write no image, and
`make test` runs the suite under the race detector. Gone are `make test` runs the suite under the race detector. Gone are
@@ -271,12 +283,11 @@ are known.
code still runs on every build (2026-10-15, code still runs on every build (2026-10-15,
https://git.eeqj.de/sneak/sfdupes/issues/95) https://git.eeqj.de/sneak/sfdupes/issues/95)
- With `-x`, an operand under another operand is no longer dropped when the - With `-x`, an operand under another operand is no longer dropped when a
outer operand's walk cannot reach it. That is the case when a directory directory between the two, the inner one included, is on a different
between the two operands, the inner one included, is on a different filesystem filesystem than the outer one. The inner operand is then walked on its own.
than the outer one. The inner operand is then walked on its own. For a For a regular-file operand the check starts at its directory. The records of
regular-file operand the directories checked start at its directory. The files gone from below it are deleted (2026-10-14,
records of files gone from below it are deleted (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/106) https://git.eeqj.de/sneak/sfdupes/issues/106)
- With `-x`, `scan` keeps the records below a directory on another filesystem - With `-x`, `scan` keeps the records below a directory on another filesystem
@@ -284,8 +295,8 @@ are known.
census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99) census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
- The hash and content phases open each file without blocking and check that it - The hash and content phases open each file without blocking and check that it
is still a regular file. A file replaced by a FIFO or device node after it was is still a regular file. A file that is now a FIFO or device node is warned
statted is warned about and skipped instead of hanging the scan (2026-10-14, about and skipped instead of hanging the scan (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/100) https://git.eeqj.de/sneak/sfdupes/issues/100)
- The scan lock file is named from the database's path with symlinks resolved, - The scan lock file is named from the database's path with symlinks resolved,
@@ -338,7 +349,7 @@ are known.
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16) (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
- `.golangci.yml` replaced with the current canonical copy, which uses - `.golangci.yml` was replaced with the current canonical copy, which uses
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04, `gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/26) https://git.eeqj.de/sneak/sfdupes/issues/26)
@@ -357,8 +368,8 @@ are known.
- A test checks that `scan` refuses a database with another schema version - A test checks that `scan` refuses a database with another schema version
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64) (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
- Four inaccurate comments in `cancel_test.go` corrected, and - Four inaccurate comments in `cancel_test.go` were corrected, and
`walkCancelInFlightDirs` renamed to `walkCancelInFlightFiles` (2026-10-04, `walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/33) https://git.eeqj.de/sneak/sfdupes/issues/33)
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04, - The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
@@ -439,18 +450,18 @@ are known.
phase. The content phase ran after the update phase. It read a file for phase. The content phase ran after the update phase. It read a file for
`content` only when its size, `head` and `tail` matched another record's, from `content` only when its size, `head` and `tail` matched another record's, from
this scan or an earlier one. `content` was the whole file below 50 MiB, and this scan or an earlier one. `content` was the whole file below 50 MiB, and
gigabyte-spaced 1 MiB samples at or above. A file gone or changed since its gigabyte-spaced 1 MiB samples at or above. It never read a file gone or
record was written was never read. `report` and `trees` left out any record changed since its record was written. `report` and `trees` left out any record
without a `content` hash. The `content` column is part of the version 1 without a `content` hash. The `content` column is part of the version 1
schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132 schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132
(2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61) (2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61)
- The dead `files.dat` references removed from `Makefile`, `.gitignore` and - The dead `files.dat` references were removed from `Makefile`, `.gitignore` and
`.dockerignore` (2026-09-21, branch `next`, `.dockerignore` (2026-09-21, branch `next`,
https://git.eeqj.de/sneak/sfdupes/issues/22) https://git.eeqj.de/sneak/sfdupes/issues/22)
- The lint-image pin comments and `FROM` form in `Dockerfile` and - The lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` fixed. Both pins became the policy `Dockerfile.lint` were fixed. Both pins became the policy
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`, `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
without the false `(Debian-based)` note or the tag. Since without the false `(Debian-based)` note or the tag. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
@@ -473,9 +484,9 @@ are known.
- The Docker build stage's prerequisites were installed by running - The Docker build stage's prerequisites were installed by running
`script/bootstrap` instead of `apk add --no-cache make` inline. `script/bootstrap` instead of `apk add --no-cache make` inline.
`script/verify-linter-pin` failed the build unless the linter copied from the `script/verify-linter-pin` failed the build unless the linter copied from the
lint stage was the version `script/bootstrap` pinned. Builds then took up to lint stage was the version that `script/bootstrap` pinned. Builds then took up
5m14s from an empty cache, mostly in a `chown -R` of the module cache, filed to 5m14s from an empty cache, mostly in a `chown -R` of the module cache,
as https://git.eeqj.de/sneak/sfdupes/issues/43. Since filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
`make` with `apk add --no-cache` and only compiles. The `lint` and `test` `make` with `apk add --no-cache` and only compiles. The `lint` and `test`
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`, phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
@@ -535,17 +546,18 @@ are known.
- the whole database layer. - the whole database layer.
The walk pool gets the same treatment, plus a `ctx.Err()` guard after the The walk pool gets the same treatment, plus a `ctx.Err()` guard after the
walk. A cancelled walk yields a partial size census, and every file it never walk. A cancelled walk yields a partial size census, and every file that it
reached looks vanished to the update phase. That phase's own `BeginTx` also never reached looks vanished to the update phase. That phase's own `BeginTx`
fails on the cancelled context before deleting anything. The guard is still also fails on the cancelled context before deleting anything. The guard is
the barrier once an interrupted scan may commit what it has. Tests drive still the barrier once an interrupted scan may commit what it has. Tests
`run(scan)` against a database whose insert trigger aborts. They assert that drive `run(scan)` against a database whose insert trigger aborts. They
the scan fails instead of hanging, and that the goroutine count returns to assert that the scan fails instead of hanging, and that the goroutine count
its pre-scan baseline. Other tests cancel a scan part-way through the walk, returns to its pre-scan baseline. Other tests cancel a scan part-way through
deterministically, by counting the scan's own checks of `ctx.Done()`. They the walk, deterministically, by counting the scan's own checks of
assert that it stops at the guard with every record intact. It then holds a `ctx.Done()`. They assert that it stops at the guard with every record
partial size census and a still-populated record index. Direct tests cover intact. It then holds a partial size census and a still-populated record
the remaining cancellation branches of both pools. They test: index. Direct tests cover the remaining cancellation branches of both pools.
They test:
- `sendEvent`; - `sendEvent`;
- the walk workers; - the walk workers;
- `dispatchDirs`; - `dispatchDirs`;
@@ -575,7 +587,7 @@ are known.
(2026-08-09, branch `db-close-on-fatal`, (2026-08-09, branch `db-close-on-fatal`,
https://git.eeqj.de/sneak/sfdupes/issues/4) https://git.eeqj.de/sneak/sfdupes/issues/4)
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as - `golangci-lint` was updated to v2.12.2 with the canonical config, merged as
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and `38a01bd`. The pinned linter in the `Dockerfile` lint stage and
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was `script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
replaced with the canonical file. Four thresholds now live under replaced with the canonical file. Four thresholds now live under
@@ -588,7 +600,7 @@ are known.
No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`, No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`,
https://git.eeqj.de/sneak/sfdupes/issues/3) https://git.eeqj.de/sneak/sfdupes/issues/3)
- `Makefile` targets converted to scripts-to-rule-them-all `script/` - `Makefile` targets were converted to scripts-to-rule-them-all `script/`
entrypoints, like the other managed repos. All 12 entrypoints exist: entrypoints, like the other managed repos. All 12 entrypoints exist:
- `bootstrap`, `setup` and `projectname`; - `bootstrap`, `setup` and `projectname`;
- `test`, `lint`, `fmt`, `fmt-check` and `check`; - `test`, `lint`, `fmt`, `fmt-check` and `check`;
@@ -601,15 +613,15 @@ are known.
it used to run `check` plus `build`. `make build` remains as an alias it used to run `check` plus `build`. `make build` remains as an alias
(2026-07-24, branch `make-default-target`) (2026-07-24, branch `make-default-target`)
- Scan-wide phases, concurrent operands and batched updates. All operands seed - The phases became scan-wide, with concurrent operands and batched updates. All
the shared walk pool, and every pass runs once over the whole scan. So the operands seed the shared walk pool, and every pass runs once over the whole
totals and ETAs cover the whole scan. The per-operand walk, hash and update scan. So the totals and ETAs cover the whole scan. The per-operand walk, hash
cycles and their stderr announcements are gone. The update pass writes in and update cycles and their stderr announcements are gone. The update pass
batches, each in its own transaction. The filesystem is authoritative and the writes in batches, each in its own transaction. The filesystem is
database follows it, so one scan's changes need not land all at once authoritative and the database follows it, so one scan's changes need not land
(2026-07-24, branch `scan-wide-phases`) all at once (2026-07-24, branch `scan-wide-phases`)
- The stat pass split back out of the walk. Phases are strictly sequential - The stat pass was split back out of the walk. Phases are strictly sequential
again, per operand, with parallelism only inside each phase: again, per operand, with parallelism only inside each phase:
- walk; - walk;
- stat; - stat;
@@ -625,31 +637,32 @@ are known.
totals that looked like the whole run's (2026-07-24, branch totals that looked like the whole run's (2026-07-24, branch
`scan-operand-progress`) `scan-operand-progress`)
- Parallel walk. The walk pass was a single goroutine and took hours at about 20 - The walk became parallel. The walk pass was a single goroutine and took hours
million files on a busy pool. One ZFS server took 4 hours for 22 million at about 20 million files on a busy pool. One ZFS server took 4 hours for 22
files. It is now a walk by a pool of per-directory workers that records size million files. It is now a walk by a pool of per-directory workers that
and mtime as it goes. That folds away the separate stat pass and halves the records size and mtime as it goes. That folds away the separate stat pass and
metadata reads. Each `PATH` operand commits in its own transaction, so an halves the metadata reads. Each `PATH` operand commits in its own transaction,
interrupted scan keeps completed operands (2026-07-24, branch `parallel-walk`) so an interrupted scan keeps completed operands (2026-07-24, branch
`parallel-walk`)
- Persistent scan database. `scan` maintains an SQLite database that survives - `scan` gained a persistent database. It maintains an SQLite database that
between runs, with the absolute path as each record's key. The driver is survives between runs, with the absolute path as each record's key. The driver
`modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only new is `modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only
or changed files, by mtime and size. It deletes the records of files vanished new or changed files, by mtime and size. It deletes the records of files
from under the scanned operands and leaves records outside them untouched. So vanished from under the scanned operands and leaves records outside them
`scan` can run daily from `cron`. `report` and `trees` read the database, with untouched. So `scan` can run daily from `cron`. `report` and `trees` read the
no positional arguments, instead of a scan stream. The database is at database, with no positional arguments, instead of a scan stream. The database
`/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL is at `/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
journaling plus a single-transaction update keep a report run during a scan journaling plus a single-transaction update keep a report run during a scan
safe (2026-07-24, branch `persistent-database`) safe (2026-07-24, branch `persistent-database`)
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, added. `v0.0.1` - The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, was added. `v0.0.1`
tagged, and `main` and the tags pushed (2026-07-23) was tagged, and `main` and the tags were pushed (2026-07-23)
- `scan` command-line rework. Required `PATH...` operands, parsed by cobra, - The `scan` command line was reworked. Required `PATH...` operands, parsed by
replace the `/srv` `-root` default. The new `-x`, `--one-file-system` flag cobra, replace the `/srv` `-root` default. The new `-x`, `--one-file-system`
follows the GNU convention and stops at filesystem boundaries, which the walk flag follows the GNU convention and stops at filesystem boundaries, which the
crosses by default (2026-07-23, branch `scan-required-paths`) walk crosses by default (2026-07-23, branch `scan-required-paths`)
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of - The repo was brought into compliance with `REPO_POLICIES.md` (the version of
2026-07-06), the existing repo checklist and the Go styleguide. The audit of 2026-07-06), the existing repo checklist and the Go styleguide. The audit of
@@ -661,8 +674,8 @@ are known.
- `LICENSE` and the README License section. MIT was assumed from house - `LICENSE` and the README License section. MIT was assumed from house
convention, for the owner to confirm; convention, for the owner to confirm;
- `REPO_POLICIES.md` in the repo root; - `REPO_POLICIES.md` in the repo root;
- the canonical `.golangci.yml`, with the 150 findings it raised fixed, so - the canonical `.golangci.yml`, with its 150 findings fixed, so that
that `make lint` is clean; `make lint` is clean;
- the `Makefile` targets the policy requires: - the `Makefile` targets the policy requires:
- `test` and `lint`; - `test` and `lint`;
- `fmt` and `fmt-check`; - `fmt` and `fmt-check`;
@@ -692,10 +705,11 @@ are known.
The code was already `gofmt`-clean, so no standalone formatting commit was The code was already `gofmt`-clean, so no standalone formatting commit was
needed (2026-07-23, branch `repo-policy-compliance`) needed (2026-07-23, branch `repo-policy-compliance`)
- `git init` with a README-only first commit; the code baseline committed on - `git init` was run, with a README-only first commit. The code baseline was
`main` (2026-07-22) committed on `main` (2026-07-22)
- `scan`, `report` and `trees` subcommands implemented (pre-git history) - The `scan`, `report` and `trees` subcommands were implemented (pre-git
history)
# Future Steps # Future Steps
+122 -10
View File
@@ -928,24 +928,21 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
} }
func TestRunScanSkipsZFSOperand(t *testing.T) { func TestRunScanSkipsZFSOperand(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
zfs := filepath.Join(t.TempDir(), ".zfs") zfs := filepath.Join(t.TempDir(), ".zfs")
snapshot := filepath.Join(zfs, "snapshot", "hourly") f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
f := writeFile(t, snapshot, "f", pattern(1, 10))
// An operand beneath a .zfs directory is walked, because it is not // No scan records a file inside a .zfs directory, so the record
// itself named .zfs. // beneath the operand is written straight into the database.
scanOK(t, snapshot) path := seedDatabase(t, []scanRec{{size: 10, path: f}})
t.Setenv(databaseEnv, path)
assertOperandSkipped(t, path, zfs, ".zfs directory", f) assertOperandSkipped(t, path, zfs, ".zfs directory", f)
} }
// assertOperandSkipped scans operand alone and checks that it is skipped // assertOperandSkipped scans operand alone and checks that it is skipped
// as kind: a warning naming it, one skip in the summary, exit 0, and the // as kind: a warning naming it, one skip in the summary, exit 0, and the
// record for kept, which an earlier scan stored beneath operand, still // record for kept, stored beneath operand before this scan, still in the
// in the database at dbPath. // database at dbPath.
func assertOperandSkipped(t *testing.T, dbPath, operand, kind, func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
kept string, kept string,
) { ) {
@@ -974,6 +971,121 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
recordByPath(t, dbRecords(t, db), kept) recordByPath(t, dbRecords(t, db), kept)
} }
// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs
// directory, first alone and then under another operand. Each time it
// is warned about and counted as skipped, and nothing in it is walked.
// The other operand is scanned as usual.
func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
writeFile(t, hourly, "f", pattern(1, 10))
f := writeFile(t, dir, "f", pattern(2, 10))
warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
for _, tc := range []struct {
operands []string
summary string
}{
{
[]string{hourly},
"scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n",
},
{
[]string{dir, hourly},
"scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n",
},
} {
stderr := scanOK(t, tc.operands...)
if !strings.Contains(stderr, warning) ||
!strings.Contains(stderr, tc.summary) {
t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands,
stderr, warning, tc.summary)
}
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
}
}
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
// by the path given, counted as skipped, and not walked.
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
dir := t.TempDir()
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
snaps := filepath.Join(dir, "snaps")
err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(snaps, "daily")
stderr := scanOK(t, daily)
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n"
if !strings.Contains(stderr, warning) ||
!strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary)
}
}
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
// a symlink to a directory outside any .zfs directory. The operand is
// scanned as usual, and its file is recorded by the path given.
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
data := filepath.Join(dir, "data")
err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(data, "daily")
stderr := scanOK(t, daily)
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
}
func TestRunReportSucceeds(t *testing.T) { func TestRunReportSucceeds(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
+28 -12
View File
@@ -383,7 +383,8 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
) error { ) error {
// Types and filesystems are checked before pruning so that an // Types and filesystems are checked before pruning so that an
// operand under a dropped one is still scanned, not dropped as lying // operand under a dropped one is still scanned, not dropped as lying
// under it. // under it, and so that an operand inside a .zfs directory gets its
// warning even when it lies under another operand.
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules, roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
deviceOfPath) deviceOfPath)
@@ -411,15 +412,15 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
} }
// walkableRoots returns the operands the walk can start from: regular // walkableRoots returns the operands the walk can start from: regular
// files, and directories not named .zfs. Every other operand is warned // files, and directories not named .zfs, none of them inside a
// about, counted as skipped, and dropped. A dropped operand is no // directory named .zfs. Every other operand is warned about, counted as
// longer a root, so the records stored beneath it are treated as if it // skipped, and dropped. A dropped operand is no longer a root, so the
// had not been given: those under no other root count as outside the // records stored beneath it are treated as if it had not been given:
// roots and are not deleted as unverified. An operand on a filesystem // those under no other root count as outside the roots and are not
// in rules that is not persistent is dropped the same way, with a line // deleted as unverified. An operand on a filesystem in rules that is not
// saying so that is not a warning, and is not counted as skipped. An // persistent is dropped the same way, with a line saying so that is not
// operand that fails lstat here is kept, and the walk warns about it // a warning, and is not counted as skipped. An operand that fails lstat
// unless it no longer exists. // here is kept, and the walk warns about it unless it no longer exists.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots)) kept := make([]string, 0, len(roots))
@@ -874,9 +875,24 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
// operandWarning returns the one-line warning for an operand the walk // operandWarning returns the one-line warning for an operand the walk
// does not start from, naming the path and what it is, or "" for one it // does not start from, naming the path and what it is, or "" for one it
// does: a regular file, or a directory not named .zfs. Symlinks are // does: a regular file, or a directory not named .zfs, that is not
// never followed, including as operands. // inside a directory named .zfs, neither as given nor with its symlinks
// resolved. Symlinks are never followed, including as operands.
func operandWarning(root string, fi fs.FileInfo) string { func operandWarning(root string, fi fs.FileInfo) string {
// The walk never enters a .zfs directory, so it starts from nothing
// inside one either, not even through a symlink. An operand whose
// symlinks cannot be resolved is checked as given.
resolved, err := filepath.EvalSymlinks(root)
if err != nil {
resolved = root
}
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
root)
}
var kind string var kind string
switch mode := fi.Mode(); { switch mode := fi.Mode(); {