Author SHA1 Message Date
sneak 261b13765a Rewrite TODO.md sentence by sentence (closes #135)
check / check (push) Canceled after 0s
TODO.md is the project's own task list, judged by the same rules as
README.md: short sentences, lists as bullets, nothing a newcomer reads
twice. Every completed step is kept with its date and its issue or
branch, and every fact the old entries recorded is kept too, in short
sentences. The 2026-07-22 policy-compliance checklist is folded into
the step that did the work, and the accepted divergence from the
layout policy gets a section of its own. Workflow, Status and Next
Step keep their rules unchanged.

Model: fable-5-1
2026-10-26 11:39:35 +00:00
4 changed files with 126 additions and 277 deletions
+11 -20
View File
@@ -461,26 +461,17 @@ exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
`-x`, is on a different filesystem than that operand, because that operand's
walk does not enter such a directory and so does not reach it.
An operand is not scanned when it is:
- a symlink, which `scan` never follows, not even as an operand;
- a socket, FIFO or device node;
- at or inside a directory named `.zfs`.
An operand reached through a symlink into a `.zfs` directory is not scanned
either.
This holds even when the operand lies under another operand. `scan` prints a
one-line warning naming the path and what it is, counts it as skipped, and drops
it from the scanned operands before reading the database. An operand beneath a
symlink operand is still scanned. The records stored beneath the dropped operand
are not deleted: they are treated like any other record outside the scanned
operands. If it lies under another operand, they are under that operand instead,
and are treated like any other record there. This is not an error: a scan whose
every operand is dropped walks nothing and exits 0. An operand that is not on a
persistent filesystem (see "Rules for the walk") is dropped the same way. The
line `scan` prints for it is not a warning, though, and it is not counted as
skipped.
An operand that is a symlink (never followed, not even as an operand), socket,
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
a one-line warning naming the path and what it is, counts it as skipped, and
drops it from the scanned operands before reading the database. Another operand
beneath it is still scanned. The records stored beneath it are not deleted: they
are treated like any other record outside the scanned operands. If it lies under
another operand, they are under that operand instead, and are treated like any
other record there. This is not an error: a scan whose every operand is dropped
walks nothing and exits 0. An operand that is not on a persistent filesystem
(see "Rules for the walk") is dropped the same way. The line `scan` prints for
it is not a warning, though, and it is not counted as skipped.
`scan` synchronizes the database with the filesystem state under the scanned
operands:
+92 -106
View File
@@ -38,14 +38,8 @@ are known.
it. It leads with what the program is and how to use it. A reference section
covers every setting and every subcommand. Development and testing come last.
Every statement was checked against the code on `next`. Five comments in
`main_test.go` cite "Settings", where their rules now live. One comment each
in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README
now says (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
- an operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. `scan` warns about it, counts it as
skipped and does not walk it (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/142)
`main_test.go` cite "Settings", where their rules now live (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/135)
- A file or directory removed while the scan runs is not an error, in the walk
or in the hash phase. The scan prints no line for it and does not count it as
@@ -63,10 +57,10 @@ are known.
`trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/134)
- The hash phase gives three hashes to every file that the walk records. `head`
and `tail` are the SHA-256 of its first and last 64 KiB. `content` is the
SHA-256 of the whole file under 100 MiB, and of 1 MiB at each gigabyte from
100 MiB up. Gone are:
- Every file the walk records gets three hashes in the hash phase. `head` and
`tail` are the SHA-256 of its first and last 64 KiB. `content` is the SHA-256
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
up. Gone are:
- the 10 MiB and 50 MiB thresholds;
- the size census, which counted the sizes of every walked file and of every
record outside the operands. A file was hashed only when the census showed
@@ -79,8 +73,8 @@ are known.
with a content hash, since every record has all three (2026-10-25,
https://git.eeqj.de/sneak/sfdupes/issues/132)
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` were copied
again from `sneak/prompts` `next` at `0653579`. Two steps now come before
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` copied again
from `sneak/prompts` `next` at `0653579`. Two steps now come before
`ARG CACHEBUST`:
- the module download in the `lint`, `test` and `builder` stages of the
`Dockerfile`;
@@ -156,12 +150,12 @@ are known.
the default excludes only, and `scan --help` lists them (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/124)
- A progress display's rate covers the items completed over the last 5 seconds.
When stderr is not a terminal, it covers the items since the line before. It
falls to 0 while nothing completes. A burst after a wait no longer reads as a
steady rate of over 1,000 files a second. The time remaining still comes from
the phase's average. Once no item has completed for 10 seconds, the line shows
`stalled` and where the phase waits in place of the rate, as
- The rate a progress display shows covers the items completed over the last 5
seconds. When stderr is not a terminal, it covers the items since the line
before. It falls to 0 while nothing completes. A burst after a wait no longer
reads as a steady rate of over 1,000 files a second. The time remaining still
comes from the phase's average. Once no item has completed for 10 seconds, the
word `stalled` and where the phase waits take the rate's place, as
`stalled disk/db`. The places are:
- `disk`;
- `db`;
@@ -176,12 +170,12 @@ are known.
- the elapsed time;
- the ETA, the time remaining.
The content phase had one display for its checks and one for its reads. Both
lost those words on an 80-column terminal, at counts like those of README's
example. That happened for most elapsed times and ETAs in minutes. Every
other phase kept them whenever the elapsed time and the ETA were both in
minutes. The content phase writes its last partial batch, the records not
yet written, under a display of its own (2026-10-16,
On an 80-column terminal, at counts like those of README's example, the
content phase's displays for its checks and its reads lost those words. That
happened for most elapsed times and ETAs in minutes. Every other phase kept
them whenever the elapsed time and the ETA were both in minutes. The content
phase writes its last partial batch, the records not yet written, under a
display of its own (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/125)
- New subcommand `info` prints the database's details on stdout, one per line as
@@ -219,20 +213,14 @@ are known.
opening the database, also while it waits on a busy one. `read` counts the
records, then reads them against that count. `group` counts the directories it
groups, and `write` the groups it writes. A display's rate names what its
phase counts:
- files;
- records;
- directories;
- groups.
stdout and the summary line are unchanged. When stdout and stderr are the
same file, as one terminal is, no line of stdout shares a line with the
display. When they are not, the display keeps being drawn while stdout waits
on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113)
phase counts, such as records or groups. stdout and the summary line are
unchanged. When stdout and stderr are the same file, as one terminal is, no
line of stdout shares a line with the display. When they are not, the display
keeps being drawn while stdout waits on a slow reader (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/113)
- Each `scan` phase's progress display names the phase and its place among the
five, as `[2/5] walk`. It runs from the moment that the phase starts. It
shows:
five, as `[2/5] walk`. It runs from the moment the phase starts and shows:
- the progress;
- the elapsed time;
- the ETA, the time remaining.
@@ -257,8 +245,8 @@ are known.
"Database" says when `scan` writes each file (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/111)
- The canonical files and model scripts were copied again from `sneak/prompts`
`next` at `e3f79c4`:
- The canonical files and model scripts copied again from `sneak/prompts` `next`
at `e3f79c4`:
- `golangci-lint` v2.14.0;
- lint and test are phases of the `Dockerfile` that write no image, and
`make test` runs the suite under the race detector. Gone are
@@ -283,11 +271,12 @@ are known.
code still runs on every build (2026-10-15,
https://git.eeqj.de/sneak/sfdupes/issues/95)
- With `-x`, an operand under another operand is no longer dropped when a
directory between the two, the inner one included, is on a different
filesystem than the outer one. The inner operand is then walked on its own.
For a regular-file operand the check starts at its directory. The records of
files gone from below it are deleted (2026-10-14,
- With `-x`, an operand under another operand is no longer dropped when the
outer operand's walk cannot reach it. That is the case when a directory
between the two operands, the inner one included, is on a different filesystem
than the outer one. The inner operand is then walked on its own. For a
regular-file operand the directories checked start at its directory. The
records of files gone from below it are deleted (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/106)
- With `-x`, `scan` keeps the records below a directory on another filesystem
@@ -295,8 +284,8 @@ are known.
census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
- The hash and content phases open each file without blocking and check that it
is still a regular file. A file that is now a FIFO or device node is warned
about and skipped instead of hanging the scan (2026-10-14,
is still a regular file. A file replaced by a FIFO or device node after it was
statted is warned about and skipped instead of hanging the scan (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/100)
- The scan lock file is named from the database's path with symlinks resolved,
@@ -349,7 +338,7 @@ are known.
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
- `.golangci.yml` was replaced with the current canonical copy, which uses
- `.golangci.yml` replaced with the current canonical copy, which uses
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/26)
@@ -368,8 +357,8 @@ are known.
- A test checks that `scan` refuses a database with another schema version
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
- Four inaccurate comments in `cancel_test.go` were corrected, and
`walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles` (2026-10-04,
- Four inaccurate comments in `cancel_test.go` corrected, and
`walkCancelInFlightDirs` renamed to `walkCancelInFlightFiles` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/33)
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
@@ -450,18 +439,18 @@ are known.
phase. The content phase ran after the update phase. It read a file for
`content` only when its size, `head` and `tail` matched another record's, from
this scan or an earlier one. `content` was the whole file below 50 MiB, and
gigabyte-spaced 1 MiB samples at or above. It never read a file gone or
changed since its record was written. `report` and `trees` left out any record
gigabyte-spaced 1 MiB samples at or above. A file gone or changed since its
record was written was never read. `report` and `trees` left out any record
without a `content` hash. The `content` column is part of the version 1
schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132
(2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61)
- The dead `files.dat` references were removed from `Makefile`, `.gitignore` and
- The dead `files.dat` references removed from `Makefile`, `.gitignore` and
`.dockerignore` (2026-09-21, branch `next`,
https://git.eeqj.de/sneak/sfdupes/issues/22)
- The lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` were fixed. Both pins became the policy
`Dockerfile.lint` fixed. Both pins became the policy
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
without the false `(Debian-based)` note or the tag. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
@@ -484,9 +473,9 @@ are known.
- The Docker build stage's prerequisites were installed by running
`script/bootstrap` instead of `apk add --no-cache make` inline.
`script/verify-linter-pin` failed the build unless the linter copied from the
lint stage was the version that `script/bootstrap` pinned. Builds then took up
to 5m14s from an empty cache, mostly in a `chown -R` of the module cache,
filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
lint stage was the version `script/bootstrap` pinned. Builds then took up to
5m14s from an empty cache, mostly in a `chown -R` of the module cache, filed
as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
`make` with `apk add --no-cache` and only compiles. The `lint` and `test`
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
@@ -546,18 +535,17 @@ are known.
- the whole database layer.
The walk pool gets the same treatment, plus a `ctx.Err()` guard after the
walk. A cancelled walk yields a partial size census, and every file that it
never reached looks vanished to the update phase. That phase's own `BeginTx`
also fails on the cancelled context before deleting anything. The guard is
still the barrier once an interrupted scan may commit what it has. Tests
drive `run(scan)` against a database whose insert trigger aborts. They
assert that the scan fails instead of hanging, and that the goroutine count
returns to its pre-scan baseline. Other tests cancel a scan part-way through
the walk, deterministically, by counting the scan's own checks of
`ctx.Done()`. They assert that it stops at the guard with every record
intact. It then holds a partial size census and a still-populated record
index. Direct tests cover the remaining cancellation branches of both pools.
They test:
walk. A cancelled walk yields a partial size census, and every file it never
reached looks vanished to the update phase. That phase's own `BeginTx` also
fails on the cancelled context before deleting anything. The guard is still
the barrier once an interrupted scan may commit what it has. Tests drive
`run(scan)` against a database whose insert trigger aborts. They assert that
the scan fails instead of hanging, and that the goroutine count returns to
its pre-scan baseline. Other tests cancel a scan part-way through the walk,
deterministically, by counting the scan's own checks of `ctx.Done()`. They
assert that it stops at the guard with every record intact. It then holds a
partial size census and a still-populated record index. Direct tests cover
the remaining cancellation branches of both pools. They test:
- `sendEvent`;
- the walk workers;
- `dispatchDirs`;
@@ -587,7 +575,7 @@ are known.
(2026-08-09, branch `db-close-on-fatal`,
https://git.eeqj.de/sneak/sfdupes/issues/4)
- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
replaced with the canonical file. Four thresholds now live under
@@ -600,7 +588,7 @@ are known.
No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`,
https://git.eeqj.de/sneak/sfdupes/issues/3)
- `Makefile` targets were converted to scripts-to-rule-them-all `script/`
- `Makefile` targets converted to scripts-to-rule-them-all `script/`
entrypoints, like the other managed repos. All 12 entrypoints exist:
- `bootstrap`, `setup` and `projectname`;
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
@@ -613,15 +601,15 @@ are known.
it used to run `check` plus `build`. `make build` remains as an alias
(2026-07-24, branch `make-default-target`)
- The phases became scan-wide, with concurrent operands and batched updates. All
operands seed the shared walk pool, and every pass runs once over the whole
scan. So the totals and ETAs cover the whole scan. The per-operand walk, hash
and update cycles and their stderr announcements are gone. The update pass
writes in batches, each in its own transaction. The filesystem is
authoritative and the database follows it, so one scan's changes need not land
all at once (2026-07-24, branch `scan-wide-phases`)
- Scan-wide phases, concurrent operands and batched updates. All operands seed
the shared walk pool, and every pass runs once over the whole scan. So the
totals and ETAs cover the whole scan. The per-operand walk, hash and update
cycles and their stderr announcements are gone. The update pass writes in
batches, each in its own transaction. The filesystem is authoritative and the
database follows it, so one scan's changes need not land all at once
(2026-07-24, branch `scan-wide-phases`)
- The stat pass was split back out of the walk. Phases are strictly sequential
- The stat pass split back out of the walk. Phases are strictly sequential
again, per operand, with parallelism only inside each phase:
- walk;
- stat;
@@ -637,32 +625,31 @@ are known.
totals that looked like the whole run's (2026-07-24, branch
`scan-operand-progress`)
- The walk became parallel. The walk pass was a single goroutine and took hours
at about 20 million files on a busy pool. One ZFS server took 4 hours for 22
million files. It is now a walk by a pool of per-directory workers that
records size and mtime as it goes. That folds away the separate stat pass and
halves the metadata reads. Each `PATH` operand commits in its own transaction,
so an interrupted scan keeps completed operands (2026-07-24, branch
`parallel-walk`)
- Parallel walk. The walk pass was a single goroutine and took hours at about 20
million files on a busy pool. One ZFS server took 4 hours for 22 million
files. It is now a walk by a pool of per-directory workers that records size
and mtime as it goes. That folds away the separate stat pass and halves the
metadata reads. Each `PATH` operand commits in its own transaction, so an
interrupted scan keeps completed operands (2026-07-24, branch `parallel-walk`)
- `scan` gained a persistent database. It maintains an SQLite database that
survives between runs, with the absolute path as each record's key. The driver
is `modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only
new or changed files, by mtime and size. It deletes the records of files
vanished from under the scanned operands and leaves records outside them
untouched. So `scan` can run daily from `cron`. `report` and `trees` read the
database, with no positional arguments, instead of a scan stream. The database
is at `/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
- Persistent scan database. `scan` maintains an SQLite database that survives
between runs, with the absolute path as each record's key. The driver is
`modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only new
or changed files, by mtime and size. It deletes the records of files vanished
from under the scanned operands and leaves records outside them untouched. So
`scan` can run daily from `cron`. `report` and `trees` read the database, with
no positional arguments, instead of a scan stream. The database is at
`/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
journaling plus a single-transaction update keep a report run during a scan
safe (2026-07-24, branch `persistent-database`)
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, was added. `v0.0.1`
was tagged, and `main` and the tags were pushed (2026-07-23)
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, added. `v0.0.1`
tagged, and `main` and the tags pushed (2026-07-23)
- The `scan` command line was reworked. Required `PATH...` operands, parsed by
cobra, replace the `/srv` `-root` default. The new `-x`, `--one-file-system`
flag follows the GNU convention and stops at filesystem boundaries, which the
walk crosses by default (2026-07-23, branch `scan-required-paths`)
- `scan` command-line rework. Required `PATH...` operands, parsed by cobra,
replace the `/srv` `-root` default. The new `-x`, `--one-file-system` flag
follows the GNU convention and stops at filesystem boundaries, which the walk
crosses by default (2026-07-23, branch `scan-required-paths`)
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
@@ -674,8 +661,8 @@ are known.
- `LICENSE` and the README License section. MIT was assumed from house
convention, for the owner to confirm;
- `REPO_POLICIES.md` in the repo root;
- the canonical `.golangci.yml`, with its 150 findings fixed, so that
`make lint` is clean;
- the canonical `.golangci.yml`, with the 150 findings it raised fixed, so
that `make lint` is clean;
- the `Makefile` targets the policy requires:
- `test` and `lint`;
- `fmt` and `fmt-check`;
@@ -705,11 +692,10 @@ are known.
The code was already `gofmt`-clean, so no standalone formatting commit was
needed (2026-07-23, branch `repo-policy-compliance`)
- `git init` was run, with a README-only first commit. The code baseline was
committed on `main` (2026-07-22)
- `git init` with a README-only first commit; the code baseline committed on
`main` (2026-07-22)
- The `scan`, `report` and `trees` subcommands were implemented (pre-git
history)
- `scan`, `report` and `trees` subcommands implemented (pre-git history)
# Future Steps
+11 -123
View File
@@ -928,21 +928,24 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
}
func TestRunScanSkipsZFSOperand(t *testing.T) {
zfs := filepath.Join(t.TempDir(), ".zfs")
f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
// No scan records a file inside a .zfs directory, so the record
// beneath the operand is written straight into the database.
path := seedDatabase(t, []scanRec{{size: 10, path: f}})
path := testDBPath(t)
t.Setenv(databaseEnv, path)
zfs := filepath.Join(t.TempDir(), ".zfs")
snapshot := filepath.Join(zfs, "snapshot", "hourly")
f := writeFile(t, snapshot, "f", pattern(1, 10))
// An operand beneath a .zfs directory is walked, because it is not
// itself named .zfs.
scanOK(t, snapshot)
assertOperandSkipped(t, path, zfs, ".zfs directory", f)
}
// assertOperandSkipped scans operand alone and checks that it is skipped
// as kind: a warning naming it, one skip in the summary, exit 0, and the
// record for kept, stored beneath operand before this scan, still in the
// database at dbPath.
// record for kept, which an earlier scan stored beneath operand, still
// in the database at dbPath.
func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
kept string,
) {
@@ -971,121 +974,6 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
recordByPath(t, dbRecords(t, db), kept)
}
// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs
// directory, first alone and then under another operand. Each time it
// is warned about and counted as skipped, and nothing in it is walked.
// The other operand is scanned as usual.
func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
writeFile(t, hourly, "f", pattern(1, 10))
f := writeFile(t, dir, "f", pattern(2, 10))
warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
for _, tc := range []struct {
operands []string
summary string
}{
{
[]string{hourly},
"scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n",
},
{
[]string{dir, hourly},
"scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n",
},
} {
stderr := scanOK(t, tc.operands...)
if !strings.Contains(stderr, warning) ||
!strings.Contains(stderr, tc.summary) {
t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands,
stderr, warning, tc.summary)
}
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
}
}
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
// by the path given, counted as skipped, and not walked.
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
dir := t.TempDir()
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
snaps := filepath.Join(dir, "snaps")
err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(snaps, "daily")
stderr := scanOK(t, daily)
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n"
if !strings.Contains(stderr, warning) ||
!strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary)
}
}
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
// a symlink to a directory outside any .zfs directory. The operand is
// scanned as usual, and its file is recorded by the path given.
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
data := filepath.Join(dir, "data")
err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(data, "daily")
stderr := scanOK(t, daily)
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
}
func TestRunReportSucceeds(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
+12 -28
View File
@@ -383,8 +383,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
) error {
// Types and filesystems are checked before pruning so that an
// operand under a dropped one is still scanned, not dropped as lying
// under it, and so that an operand inside a .zfs directory gets its
// warning even when it lies under another operand.
// under it.
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
deviceOfPath)
@@ -412,15 +411,15 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
}
// walkableRoots returns the operands the walk can start from: regular
// files, and directories not named .zfs, none of them inside a
// directory named .zfs. Every other operand is warned about, counted as
// skipped, and dropped. A dropped operand is no longer a root, so the
// records stored beneath it are treated as if it had not been given:
// those under no other root count as outside the roots and are not
// deleted as unverified. An operand on a filesystem in rules that is not
// persistent is dropped the same way, with a line saying so that is not
// a warning, and is not counted as skipped. An operand that fails lstat
// here is kept, and the walk warns about it unless it no longer exists.
// files, and directories not named .zfs. Every other operand is warned
// about, counted as skipped, and dropped. A dropped operand is no
// longer a root, so the records stored beneath it are treated as if it
// had not been given: those under no other root count as outside the
// roots and are not deleted as unverified. An operand on a filesystem
// in rules that is not persistent is dropped the same way, with a line
// saying so that is not a warning, and is not counted as skipped. An
// operand that fails lstat here is kept, and the walk warns about it
// unless it no longer exists.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots))
@@ -875,24 +874,9 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
// operandWarning returns the one-line warning for an operand the walk
// does not start from, naming the path and what it is, or "" for one it
// does: a regular file, or a directory not named .zfs, that is not
// inside a directory named .zfs, neither as given nor with its symlinks
// resolved. Symlinks are never followed, including as operands.
// does: a regular file, or a directory not named .zfs. Symlinks are
// never followed, including as operands.
func operandWarning(root string, fi fs.FileInfo) string {
// The walk never enters a .zfs directory, so it starts from nothing
// inside one either, not even through a symlink. An operand whose
// symlinks cannot be resolved is checked as given.
resolved, err := filepath.EvalSymlinks(root)
if err != nil {
resolved = root
}
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
root)
}
var kind string
switch mode := fi.Mode(); {