Compare commits
1
Commits
next
..
8a33157a63
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8a33157a63 |
No files matched your search
@@ -29,63 +29,17 @@
|
|||||||
Newest first. An entry ends with its date and its issue or branch, where those
|
Newest first. An entry ends with its date and its issue or branch, where those
|
||||||
are known.
|
are known.
|
||||||
|
|
||||||
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
|
|
||||||
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
|
|
||||||
prints the report again (2026-11-06,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/154)
|
|
||||||
|
|
||||||
- `scan` resolves each operand to its real path, symlinks followed, before the
|
|
||||||
walk. A symlink operand is scanned at its target instead of skipped. With
|
|
||||||
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
|
|
||||||
walks `sub` on its own instead of dropping it and deleting its records
|
|
||||||
(2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
|
|
||||||
|
|
||||||
- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's
|
|
||||||
blocks only seconds after the write. The count can land in under a second.
|
|
||||||
Both now say up to seconds after the write, as the comment on `diskSizeOf` in
|
|
||||||
`scan_test.go` does (2026-11-03, https://git.eeqj.de/sneak/sfdupes/issues/152)
|
|
||||||
|
|
||||||
- The three disk size tests in `scan_test.go` no longer fail at random on ZFS.
|
|
||||||
`unix.Sync` can return before ZFS has counted a file's blocks, and one ZFS
|
|
||||||
commit can take a file through several sizes. The test that needs the size to
|
|
||||||
change waits for it, for at most 30 seconds. Each test reads the size before
|
|
||||||
and after its scan. The rewrite test rescans until the two reads match, then
|
|
||||||
compares exactly. The other two fail only on a `disk_size` outside the two
|
|
||||||
reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
|
|
||||||
|
|
||||||
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
|
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
|
||||||
sentence. Every completed step is kept, with its date and its issue or branch.
|
sentence. Every completed step is kept, with its date and its issue or branch.
|
||||||
The policy-compliance checklist of 2026-07-22 is folded into its step
|
The policy-compliance checklist of 2026-07-22 is folded into its step
|
||||||
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
(2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||||
|
|
||||||
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
|
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
|
||||||
it. It leads with what the program is and how to use it. A reference section
|
it. It leads with what the program is and how to use it. A reference section
|
||||||
covers every setting and every subcommand. Development and testing come last.
|
covers every setting and every subcommand. Development and testing come last.
|
||||||
Every statement was checked against the code on `next`. Five comments in
|
Every statement was checked against the code on `next`. Three comments in
|
||||||
`main_test.go` cite "Settings", where their rules now live. One comment each
|
`main_test.go` cite "Settings", where their rules now live (2026-10-26,
|
||||||
in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README
|
https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||||
now says (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
|
||||||
|
|
||||||
- Each record holds `disk_size`, the bytes the file takes on disk. That is its
|
|
||||||
allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already
|
|
||||||
makes. `trees` prints a `disk_size` column after `size`. It is the sum over
|
|
||||||
the `dupe` copy's files. On a compressed or sparse file `disk_size` differs
|
|
||||||
from `size`. A rescan updates the `disk_size` of an unchanged file whose size
|
|
||||||
on disk changed, without reading the file (2026-10-27,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/147)
|
|
||||||
|
|
||||||
- The database keeps the number of records and the sum of their sizes in the one
|
|
||||||
row of a `totals` table. Three triggers on `files` keep that row current.
|
|
||||||
`info` reads that row. So do `scan`'s `load` phase and the `read` phases of
|
|
||||||
`report` and `trees`, for their totals. No count reads every record any more.
|
|
||||||
The triggers make each write to `files` slower. `scan` keeps SQLite's
|
|
||||||
temporary data in memory. That wins back part of that time (2026-10-27,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/146)
|
|
||||||
|
|
||||||
- an operand at or inside a directory named `.zfs` is never scanned, whether
|
|
||||||
given alone or under another operand. `scan` warns about it, counts it as
|
|
||||||
skipped and does not walk it (2026-10-26,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/142)
|
|
||||||
|
|
||||||
- A file or directory removed while the scan runs is not an error, in the walk
|
- A file or directory removed while the scan runs is not an error, in the walk
|
||||||
or in the hash phase. The scan prints no line for it and does not count it as
|
or in the hash phase. The scan prints no line for it and does not count it as
|
||||||
@@ -103,10 +57,10 @@ are known.
|
|||||||
`trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26,
|
`trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/134)
|
https://git.eeqj.de/sneak/sfdupes/issues/134)
|
||||||
|
|
||||||
- The hash phase gives three hashes to every file that the walk records. `head`
|
- Every file the walk records gets three hashes in the hash phase. `head` and
|
||||||
and `tail` are the SHA-256 of its first and last 64 KiB. `content` is the
|
`tail` are the SHA-256 of its first and last 64 KiB. `content` is the SHA-256
|
||||||
SHA-256 of the whole file under 100 MiB, and of 1 MiB at each gigabyte from
|
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
|
||||||
100 MiB up. Gone are:
|
up. Gone are:
|
||||||
- the 10 MiB and 50 MiB thresholds;
|
- the 10 MiB and 50 MiB thresholds;
|
||||||
- the size census, which counted the sizes of every walked file and of every
|
- the size census, which counted the sizes of every walked file and of every
|
||||||
record outside the operands. A file was hashed only when the census showed
|
record outside the operands. A file was hashed only when the census showed
|
||||||
@@ -119,8 +73,8 @@ are known.
|
|||||||
with a content hash, since every record has all three (2026-10-25,
|
with a content hash, since every record has all three (2026-10-25,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/132)
|
https://git.eeqj.de/sneak/sfdupes/issues/132)
|
||||||
|
|
||||||
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` were copied
|
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` copied again
|
||||||
again from `sneak/prompts` `next` at `0653579`. Two steps now come before
|
from `sneak/prompts` `next` at `0653579`. Two steps now come before
|
||||||
`ARG CACHEBUST`:
|
`ARG CACHEBUST`:
|
||||||
- the module download in the `lint`, `test` and `builder` stages of the
|
- the module download in the `lint`, `test` and `builder` stages of the
|
||||||
`Dockerfile`;
|
`Dockerfile`;
|
||||||
@@ -136,13 +90,12 @@ are known.
|
|||||||
README says the same (2026-10-18,
|
README says the same (2026-10-18,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/109)
|
https://git.eeqj.de/sneak/sfdupes/issues/109)
|
||||||
|
|
||||||
- The release rule was set. Until 1.0, each tip of `main` that passes the build
|
- Release rule set. Until 1.0, each tip of `main` that passes the build gets the
|
||||||
gets the next `v0.0.N` tag. `go install sneak.berlin/go/sfdupes@latest` then
|
next `v0.0.N` tag. `go install sneak.berlin/go/sfdupes@latest` then installs
|
||||||
installs it. `go.mod` retracts `v0.0.1`, the early build without a database.
|
it. `go.mod` retracts `v0.0.1`, the early build without a database. When the
|
||||||
When the linker set no version, `sfdupes --version` prints the version Go
|
linker set no version, `sfdupes --version` prints the version Go recorded in
|
||||||
recorded in the binary. That is the tag, or the pseudo-version of a
|
the binary. That is the tag, or the pseudo-version of a `go install` build. It
|
||||||
`go install` build. It prints `dev` only when neither is known. README
|
prints `dev` only when neither is known. README "Install" documented:
|
||||||
"Install" documented:
|
|
||||||
- the rule;
|
- the rule;
|
||||||
- what `@main` and `v0.0.1` give;
|
- what `@main` and `v0.0.1` give;
|
||||||
- how to check an installed binary.
|
- how to check an installed binary.
|
||||||
@@ -196,12 +149,12 @@ are known.
|
|||||||
the default excludes only, and `scan --help` lists them (2026-10-17,
|
the default excludes only, and `scan --help` lists them (2026-10-17,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
||||||
|
|
||||||
- A progress display's rate covers the items completed over the last 5 seconds.
|
- The rate a progress display shows covers the items completed over the last 5
|
||||||
When stderr is not a terminal, it covers the items since the line before. It
|
seconds. When stderr is not a terminal, it covers the items since the line
|
||||||
falls to 0 while nothing completes. A burst after a wait no longer reads as a
|
before. It falls to 0 while nothing completes. A burst after a wait no longer
|
||||||
steady rate of over 1,000 files a second. The time remaining still comes from
|
reads as a steady rate of over 1,000 files a second. The time remaining still
|
||||||
the phase's average. Once no item has completed for 10 seconds, the line shows
|
comes from the phase's average. Once no item has completed for 10 seconds, the
|
||||||
`stalled` and where the phase waits in place of the rate, as
|
word `stalled` and where the phase waits take the rate's place, as
|
||||||
`stalled disk/db`. The places are:
|
`stalled disk/db`. The places are:
|
||||||
- `disk`;
|
- `disk`;
|
||||||
- `db`;
|
- `db`;
|
||||||
@@ -216,12 +169,12 @@ are known.
|
|||||||
- the elapsed time;
|
- the elapsed time;
|
||||||
- the ETA, the time remaining.
|
- the ETA, the time remaining.
|
||||||
|
|
||||||
The content phase had one display for its checks and one for its reads. Both
|
On an 80-column terminal, at counts like those of README's example, the
|
||||||
lost those words on an 80-column terminal, at counts like those of README's
|
content phase's displays for its checks and its reads lost those words. That
|
||||||
example. That happened for most elapsed times and ETAs in minutes. Every
|
happened for most elapsed times and ETAs in minutes. Every other phase kept
|
||||||
other phase kept them whenever the elapsed time and the ETA were both in
|
them whenever the elapsed time and the ETA were both in minutes. The content
|
||||||
minutes. The content phase writes its last partial batch, the records not
|
phase writes its last partial batch, the records not yet written, under a
|
||||||
yet written, under a display of its own (2026-10-16,
|
display of its own (2026-10-16,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||||
|
|
||||||
- New subcommand `info` prints the database's details on stdout, one per line as
|
- New subcommand `info` prints the database's details on stdout, one per line as
|
||||||
@@ -259,20 +212,14 @@ are known.
|
|||||||
opening the database, also while it waits on a busy one. `read` counts the
|
opening the database, also while it waits on a busy one. `read` counts the
|
||||||
records, then reads them against that count. `group` counts the directories it
|
records, then reads them against that count. `group` counts the directories it
|
||||||
groups, and `write` the groups it writes. A display's rate names what its
|
groups, and `write` the groups it writes. A display's rate names what its
|
||||||
phase counts:
|
phase counts, such as records or groups. stdout and the summary line are
|
||||||
- files;
|
unchanged. When stdout and stderr are the same file, as one terminal is, no
|
||||||
- records;
|
line of stdout shares a line with the display. When they are not, the display
|
||||||
- directories;
|
keeps being drawn while stdout waits on a slow reader (2026-10-16,
|
||||||
- groups.
|
https://git.eeqj.de/sneak/sfdupes/issues/113)
|
||||||
|
|
||||||
stdout and the summary line are unchanged. When stdout and stderr are the
|
|
||||||
same file, as one terminal is, no line of stdout shares a line with the
|
|
||||||
display. When they are not, the display keeps being drawn while stdout waits
|
|
||||||
on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113)
|
|
||||||
|
|
||||||
- Each `scan` phase's progress display names the phase and its place among the
|
- Each `scan` phase's progress display names the phase and its place among the
|
||||||
five, as `[2/5] walk`. It runs from the moment that the phase starts. It
|
five, as `[2/5] walk`. It runs from the moment the phase starts and shows:
|
||||||
shows:
|
|
||||||
- the progress;
|
- the progress;
|
||||||
- the elapsed time;
|
- the elapsed time;
|
||||||
- the ETA, the time remaining.
|
- the ETA, the time remaining.
|
||||||
@@ -297,8 +244,8 @@ are known.
|
|||||||
"Database" says when `scan` writes each file (2026-10-16,
|
"Database" says when `scan` writes each file (2026-10-16,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/111)
|
https://git.eeqj.de/sneak/sfdupes/issues/111)
|
||||||
|
|
||||||
- The canonical files and model scripts were copied again from `sneak/prompts`
|
- The canonical files and model scripts copied again from `sneak/prompts` `next`
|
||||||
`next` at `e3f79c4`:
|
at `e3f79c4`:
|
||||||
- `golangci-lint` v2.14.0;
|
- `golangci-lint` v2.14.0;
|
||||||
- lint and test are phases of the `Dockerfile` that write no image, and
|
- lint and test are phases of the `Dockerfile` that write no image, and
|
||||||
`make test` runs the suite under the race detector. Gone are
|
`make test` runs the suite under the race detector. Gone are
|
||||||
@@ -312,7 +259,7 @@ are known.
|
|||||||
gone, and the build stage installs `git` and `make` itself;
|
gone, and the build stage installs `git` and `make` itself;
|
||||||
- a new push cancels the workflow's older run on the same branch, and a run
|
- a new push cancels the workflow's older run on the same branch, and a run
|
||||||
stops after 20 minutes;
|
stops after 20 minutes;
|
||||||
- a coding agent's settings file is deleted.
|
- a stray editor-settings file is deleted.
|
||||||
|
|
||||||
Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before
|
Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before
|
||||||
`ARG CACHEBUST`:
|
`ARG CACHEBUST`:
|
||||||
@@ -323,11 +270,12 @@ are known.
|
|||||||
code still runs on every build (2026-10-15,
|
code still runs on every build (2026-10-15,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95)
|
https://git.eeqj.de/sneak/sfdupes/issues/95)
|
||||||
|
|
||||||
- With `-x`, an operand under another operand is no longer dropped when a
|
- With `-x`, an operand under another operand is no longer dropped when the
|
||||||
directory between the two, the inner one included, is on a different
|
outer operand's walk cannot reach it. That is the case when a directory
|
||||||
filesystem than the outer one. The inner operand is then walked on its own.
|
between the two operands, the inner one included, is on a different filesystem
|
||||||
For a regular-file operand the check starts at its directory. The records of
|
than the outer one. The inner operand is then walked on its own. For a
|
||||||
files gone from below it are deleted (2026-10-14,
|
regular-file operand the directories checked start at its directory. The
|
||||||
|
records of files gone from below it are deleted (2026-10-14,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/106)
|
https://git.eeqj.de/sneak/sfdupes/issues/106)
|
||||||
|
|
||||||
- With `-x`, `scan` keeps the records below a directory on another filesystem
|
- With `-x`, `scan` keeps the records below a directory on another filesystem
|
||||||
@@ -335,8 +283,8 @@ are known.
|
|||||||
census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
|
census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
|
||||||
|
|
||||||
- The hash and content phases open each file without blocking and check that it
|
- The hash and content phases open each file without blocking and check that it
|
||||||
is still a regular file. A file that is now a FIFO or device node is warned
|
is still a regular file. A file replaced by a FIFO or device node after it was
|
||||||
about and skipped instead of hanging the scan (2026-10-14,
|
statted is warned about and skipped instead of hanging the scan (2026-10-14,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/100)
|
https://git.eeqj.de/sneak/sfdupes/issues/100)
|
||||||
|
|
||||||
- The scan lock file is named from the database's path with symlinks resolved,
|
- The scan lock file is named from the database's path with symlinks resolved,
|
||||||
@@ -373,8 +321,8 @@ are known.
|
|||||||
`script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04,
|
`script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/39)
|
https://git.eeqj.de/sneak/sfdupes/issues/39)
|
||||||
|
|
||||||
- `make fmt` and `make fmt-check` run `prettier` over all Markdown, and the
|
- `make fmt` and `make fmt-check` run `prettier` over all Markdown, and CI
|
||||||
Gitea workflow checks it. All Markdown was reformatted (2026-10-04,
|
checks it. All Markdown was reformatted (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/19)
|
https://git.eeqj.de/sneak/sfdupes/issues/19)
|
||||||
|
|
||||||
- `script/lint` writes no image, so a run no longer leaves an untagged one
|
- `script/lint` writes no image, so a run no longer leaves an untagged one
|
||||||
@@ -389,7 +337,7 @@ are known.
|
|||||||
|
|
||||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
||||||
|
|
||||||
- `.golangci.yml` was replaced with the current canonical copy, which uses
|
- `.golangci.yml` replaced with the current canonical copy, which uses
|
||||||
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/26)
|
https://git.eeqj.de/sneak/sfdupes/issues/26)
|
||||||
|
|
||||||
@@ -408,8 +356,8 @@ are known.
|
|||||||
- A test checks that `scan` refuses a database with another schema version
|
- A test checks that `scan` refuses a database with another schema version
|
||||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
|
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
|
||||||
|
|
||||||
- Four inaccurate comments in `cancel_test.go` were corrected, and
|
- Four inaccurate comments in `cancel_test.go` corrected, and
|
||||||
`walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles` (2026-10-04,
|
`walkCancelInFlightDirs` renamed to `walkCancelInFlightFiles` (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/33)
|
https://git.eeqj.de/sneak/sfdupes/issues/33)
|
||||||
|
|
||||||
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
|
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
|
||||||
@@ -479,8 +427,8 @@ are known.
|
|||||||
`dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage
|
`dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage
|
||||||
stamps the `VERSION` build argument, else `git describe --tags --always`. It
|
stamps the `VERSION` build argument, else `git describe --tags --always`. It
|
||||||
fails if the context carries `.git` and the version is still empty, `dev` or
|
fails if the context carries `.git` and the version is still empty, `dev` or
|
||||||
`unknown`. The Gitea workflow checks out the full history (`fetch-depth: 0`),
|
`unknown`. CI checks out the full history (`fetch-depth: 0`), so it stamps the
|
||||||
so it stamps the same value as `make build` (2026-10-02, branch `next`,
|
same value as `make build` (2026-10-02, branch `next`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/67)
|
https://git.eeqj.de/sneak/sfdupes/issues/67)
|
||||||
|
|
||||||
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
|
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
|
||||||
@@ -490,18 +438,18 @@ are known.
|
|||||||
phase. The content phase ran after the update phase. It read a file for
|
phase. The content phase ran after the update phase. It read a file for
|
||||||
`content` only when its size, `head` and `tail` matched another record's, from
|
`content` only when its size, `head` and `tail` matched another record's, from
|
||||||
this scan or an earlier one. `content` was the whole file below 50 MiB, and
|
this scan or an earlier one. `content` was the whole file below 50 MiB, and
|
||||||
gigabyte-spaced 1 MiB samples at or above. It never read a file gone or
|
gigabyte-spaced 1 MiB samples at or above. A file gone or changed since its
|
||||||
changed since its record was written. `report` and `trees` left out any record
|
record was written was never read. `report` and `trees` left out any record
|
||||||
without a `content` hash. The `content` column is part of the version 1
|
without a `content` hash. The `content` column is part of the version 1
|
||||||
schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132
|
schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132
|
||||||
(2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61)
|
(2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61)
|
||||||
|
|
||||||
- The dead `files.dat` references were removed from `Makefile`, `.gitignore` and
|
- The dead `files.dat` references removed from `Makefile`, `.gitignore` and
|
||||||
`.dockerignore` (2026-09-21, branch `next`,
|
`.dockerignore` (2026-09-21, branch `next`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/22)
|
https://git.eeqj.de/sneak/sfdupes/issues/22)
|
||||||
|
|
||||||
- The lint-image pin comments and `FROM` form in `Dockerfile` and
|
- The lint-image pin comments and `FROM` form in `Dockerfile` and
|
||||||
`Dockerfile.lint` were fixed. Both pins became the policy
|
`Dockerfile.lint` fixed. Both pins became the policy
|
||||||
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
|
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
|
||||||
without the false `(Debian-based)` note or the tag. Since
|
without the false `(Debian-based)` note or the tag. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
|
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
|
||||||
@@ -524,9 +472,9 @@ are known.
|
|||||||
- The Docker build stage's prerequisites were installed by running
|
- The Docker build stage's prerequisites were installed by running
|
||||||
`script/bootstrap` instead of `apk add --no-cache make` inline.
|
`script/bootstrap` instead of `apk add --no-cache make` inline.
|
||||||
`script/verify-linter-pin` failed the build unless the linter copied from the
|
`script/verify-linter-pin` failed the build unless the linter copied from the
|
||||||
lint stage was the version that `script/bootstrap` pinned. Builds then took up
|
lint stage was the version `script/bootstrap` pinned. Builds then took up to
|
||||||
to 5m14s from an empty cache, mostly in a `chown -R` of the module cache,
|
5m14s from an empty cache, mostly in a `chown -R` of the module cache, filed
|
||||||
filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
|
as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
|
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
|
||||||
`make` with `apk add --no-cache` and only compiles. The `lint` and `test`
|
`make` with `apk add --no-cache` and only compiles. The `lint` and `test`
|
||||||
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
|
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
|
||||||
@@ -586,18 +534,17 @@ are known.
|
|||||||
- the whole database layer.
|
- the whole database layer.
|
||||||
|
|
||||||
The walk pool gets the same treatment, plus a `ctx.Err()` guard after the
|
The walk pool gets the same treatment, plus a `ctx.Err()` guard after the
|
||||||
walk. A cancelled walk yields a partial size census, and every file that it
|
walk. A cancelled walk yields a partial size census, and every file it never
|
||||||
never reached looks vanished to the update phase. That phase's own `BeginTx`
|
reached looks vanished to the update phase. That phase's own `BeginTx` also
|
||||||
also fails on the cancelled context before deleting anything. The guard is
|
fails on the cancelled context before deleting anything. The guard is still
|
||||||
still the barrier once an interrupted scan may commit what it has. Tests
|
the barrier once an interrupted scan may commit what it has. Tests drive
|
||||||
drive `run(scan)` against a database whose insert trigger aborts. They
|
`run(scan)` against a database whose insert trigger aborts. They assert that
|
||||||
assert that the scan fails instead of hanging, and that the goroutine count
|
the scan fails instead of hanging, and that the goroutine count returns to
|
||||||
returns to its pre-scan baseline. Other tests cancel a scan part-way through
|
its pre-scan baseline. Other tests cancel a scan part-way through the walk,
|
||||||
the walk, deterministically, by counting the scan's own checks of
|
deterministically, by counting the scan's own checks of `ctx.Done()`. They
|
||||||
`ctx.Done()`. They assert that it stops at the guard with every record
|
assert that it stops at the guard with every record intact. It then holds a
|
||||||
intact. It then holds a partial size census and a still-populated record
|
partial size census and a still-populated record index. Direct tests cover
|
||||||
index. Direct tests cover the remaining cancellation branches of both pools.
|
the remaining cancellation branches of both pools. They test:
|
||||||
They test:
|
|
||||||
- `sendEvent`;
|
- `sendEvent`;
|
||||||
- the walk workers;
|
- the walk workers;
|
||||||
- `dispatchDirs`;
|
- `dispatchDirs`;
|
||||||
@@ -627,7 +574,7 @@ are known.
|
|||||||
(2026-08-09, branch `db-close-on-fatal`,
|
(2026-08-09, branch `db-close-on-fatal`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||||
|
|
||||||
- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as
|
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as
|
||||||
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
|
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
|
||||||
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
|
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
|
||||||
replaced with the canonical file. Four thresholds now live under
|
replaced with the canonical file. Four thresholds now live under
|
||||||
@@ -640,7 +587,7 @@ are known.
|
|||||||
No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`,
|
No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/3)
|
https://git.eeqj.de/sneak/sfdupes/issues/3)
|
||||||
|
|
||||||
- `Makefile` targets were converted to scripts-to-rule-them-all `script/`
|
- `Makefile` targets converted to scripts-to-rule-them-all `script/`
|
||||||
entrypoints, like the other managed repos. All 12 entrypoints exist:
|
entrypoints, like the other managed repos. All 12 entrypoints exist:
|
||||||
- `bootstrap`, `setup` and `projectname`;
|
- `bootstrap`, `setup` and `projectname`;
|
||||||
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
|
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
|
||||||
@@ -653,15 +600,15 @@ are known.
|
|||||||
it used to run `check` plus `build`. `make build` remains as an alias
|
it used to run `check` plus `build`. `make build` remains as an alias
|
||||||
(2026-07-24, branch `make-default-target`)
|
(2026-07-24, branch `make-default-target`)
|
||||||
|
|
||||||
- The phases became scan-wide, with concurrent operands and batched updates. All
|
- Scan-wide phases, concurrent operands and batched updates. All operands seed
|
||||||
operands seed the shared walk pool, and every pass runs once over the whole
|
the shared walk pool, and every pass runs once over the whole scan. So the
|
||||||
scan. So the totals and ETAs cover the whole scan. The per-operand walk, hash
|
totals and ETAs cover the whole scan. The per-operand walk, hash and update
|
||||||
and update cycles and their stderr announcements are gone. The update pass
|
cycles and their stderr announcements are gone. The update pass writes in
|
||||||
writes in batches, each in its own transaction. The filesystem is
|
batches, each in its own transaction. The filesystem is authoritative and the
|
||||||
authoritative and the database follows it, so one scan's changes need not land
|
database follows it, so one scan's changes need not land all at once
|
||||||
all at once (2026-07-24, branch `scan-wide-phases`)
|
(2026-07-24, branch `scan-wide-phases`)
|
||||||
|
|
||||||
- The stat pass was split back out of the walk. Phases are strictly sequential
|
- The stat pass split back out of the walk. Phases are strictly sequential
|
||||||
again, per operand, with parallelism only inside each phase:
|
again, per operand, with parallelism only inside each phase:
|
||||||
- walk;
|
- walk;
|
||||||
- stat;
|
- stat;
|
||||||
@@ -677,32 +624,31 @@ are known.
|
|||||||
totals that looked like the whole run's (2026-07-24, branch
|
totals that looked like the whole run's (2026-07-24, branch
|
||||||
`scan-operand-progress`)
|
`scan-operand-progress`)
|
||||||
|
|
||||||
- The walk became parallel. The walk pass was a single goroutine and took hours
|
- Parallel walk. The walk pass was a single goroutine and took hours at about 20
|
||||||
at about 20 million files on a busy pool. One ZFS server took 4 hours for 22
|
million files on a busy pool. One ZFS server took 4 hours for 22 million
|
||||||
million files. It is now a walk by a pool of per-directory workers that
|
files. It is now a walk by a pool of per-directory workers that records size
|
||||||
records size and mtime as it goes. That folds away the separate stat pass and
|
and mtime as it goes. That folds away the separate stat pass and halves the
|
||||||
halves the metadata reads. Each `PATH` operand commits in its own transaction,
|
metadata reads. Each `PATH` operand commits in its own transaction, so an
|
||||||
so an interrupted scan keeps completed operands (2026-07-24, branch
|
interrupted scan keeps completed operands (2026-07-24, branch `parallel-walk`)
|
||||||
`parallel-walk`)
|
|
||||||
|
|
||||||
- `scan` gained a persistent database. It maintains an SQLite database that
|
- Persistent scan database. `scan` maintains an SQLite database that survives
|
||||||
survives between runs, with the absolute path as each record's key. The driver
|
between runs, with the absolute path as each record's key. The driver is
|
||||||
is `modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only
|
`modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only new
|
||||||
new or changed files, by mtime and size. It deletes the records of files
|
or changed files, by mtime and size. It deletes the records of files vanished
|
||||||
vanished from under the scanned operands and leaves records outside them
|
from under the scanned operands and leaves records outside them untouched. So
|
||||||
untouched. So `scan` can run daily from `cron`. `report` and `trees` read the
|
`scan` can run daily from `cron`. `report` and `trees` read the database, with
|
||||||
database, with no positional arguments, instead of a scan stream. The database
|
no positional arguments, instead of a scan stream. The database is at
|
||||||
is at `/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
|
`/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
|
||||||
journaling plus a single-transaction update keep a report run during a scan
|
journaling plus a single-transaction update keep a report run during a scan
|
||||||
safe (2026-07-24, branch `persistent-database`)
|
safe (2026-07-24, branch `persistent-database`)
|
||||||
|
|
||||||
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, was added. `v0.0.1`
|
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, added. `v0.0.1`
|
||||||
was tagged, and `main` and the tags were pushed (2026-07-23)
|
tagged, and `main` and the tags pushed (2026-07-23)
|
||||||
|
|
||||||
- The `scan` command line was reworked. Required `PATH...` operands, parsed by
|
- `scan` command-line rework. Required `PATH...` operands, parsed by cobra,
|
||||||
cobra, replace the `/srv` `-root` default. The new `-x`, `--one-file-system`
|
replace the `/srv` `-root` default. The new `-x`, `--one-file-system` flag
|
||||||
flag follows the GNU convention and stops at filesystem boundaries, which the
|
follows the GNU convention and stops at filesystem boundaries, which the walk
|
||||||
walk crosses by default (2026-07-23, branch `scan-required-paths`)
|
crosses by default (2026-07-23, branch `scan-required-paths`)
|
||||||
|
|
||||||
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
|
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
|
||||||
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
|
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
|
||||||
@@ -714,8 +660,8 @@ are known.
|
|||||||
- `LICENSE` and the README License section. MIT was assumed from house
|
- `LICENSE` and the README License section. MIT was assumed from house
|
||||||
convention, for the owner to confirm;
|
convention, for the owner to confirm;
|
||||||
- `REPO_POLICIES.md` in the repo root;
|
- `REPO_POLICIES.md` in the repo root;
|
||||||
- the canonical `.golangci.yml`, with its 150 findings fixed, so that
|
- the canonical `.golangci.yml`, with the 150 findings it raised fixed, so
|
||||||
`make lint` is clean;
|
that `make lint` is clean;
|
||||||
- the `Makefile` targets the policy requires:
|
- the `Makefile` targets the policy requires:
|
||||||
- `test` and `lint`;
|
- `test` and `lint`;
|
||||||
- `fmt` and `fmt-check`;
|
- `fmt` and `fmt-check`;
|
||||||
@@ -745,11 +691,10 @@ are known.
|
|||||||
The code was already `gofmt`-clean, so no standalone formatting commit was
|
The code was already `gofmt`-clean, so no standalone formatting commit was
|
||||||
needed (2026-07-23, branch `repo-policy-compliance`)
|
needed (2026-07-23, branch `repo-policy-compliance`)
|
||||||
|
|
||||||
- `git init` was run, with a README-only first commit. The code baseline was
|
- `git init` with a README-only first commit; the code baseline committed on
|
||||||
committed on `main` (2026-07-22)
|
`main` (2026-07-22)
|
||||||
|
|
||||||
- The `scan`, `report` and `trees` subcommands were implemented (pre-git
|
- `scan`, `report` and `trees` subcommands implemented (pre-git history)
|
||||||
history)
|
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
|
|||||||
@@ -47,15 +47,14 @@ const dbFilePerm = 0o644
|
|||||||
const lockFilePerm = 0o600
|
const lockFilePerm = 0o600
|
||||||
|
|
||||||
// createTableSQL is the schema applied to a fresh database. Paths are
|
// createTableSQL is the schema applied to a fresh database. Paths are
|
||||||
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
|
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime
|
||||||
// disk_size holds the bytes the file takes on disk. mtime holds whole
|
// holds whole Unix seconds and mtime_nsec the nanoseconds within that
|
||||||
// Unix seconds and mtime_nsec the nanoseconds within that second. head,
|
// second. head, tail and content each hold the raw 32 bytes of a
|
||||||
// tail and content each hold the raw 32 bytes of a SHA-256.
|
// SHA-256.
|
||||||
const createTableSQL = `
|
const createTableSQL = `
|
||||||
CREATE TABLE files (
|
CREATE TABLE files (
|
||||||
path BLOB PRIMARY KEY,
|
path BLOB PRIMARY KEY,
|
||||||
size INTEGER NOT NULL,
|
size INTEGER NOT NULL,
|
||||||
disk_size INTEGER NOT NULL,
|
|
||||||
mtime INTEGER NOT NULL,
|
mtime INTEGER NOT NULL,
|
||||||
mtime_nsec INTEGER NOT NULL,
|
mtime_nsec INTEGER NOT NULL,
|
||||||
head BLOB NOT NULL,
|
head BLOB NOT NULL,
|
||||||
@@ -70,59 +69,14 @@ const createIndexSQL = `
|
|||||||
CREATE INDEX files_signature ON files (size, head, tail, content)
|
CREATE INDEX files_signature ON files (size, head, tail, content)
|
||||||
`
|
`
|
||||||
|
|
||||||
// createTotalsSQL creates the totals table, which holds one row: the
|
|
||||||
// number of records and the sizes of all records added up. Counting
|
|
||||||
// them over files would read every record. The three triggers below
|
|
||||||
// keep the row current, in the transaction of each change to files.
|
|
||||||
const createTotalsSQL = `
|
|
||||||
CREATE TABLE totals (
|
|
||||||
records INTEGER NOT NULL,
|
|
||||||
size INTEGER NOT NULL
|
|
||||||
)
|
|
||||||
`
|
|
||||||
|
|
||||||
// insertTotalsSQL adds the one row of totals, for an empty files table.
|
|
||||||
const insertTotalsSQL = `
|
|
||||||
INSERT INTO totals (records, size) VALUES (0, 0)
|
|
||||||
`
|
|
||||||
|
|
||||||
// totalsAfterInsertSQL counts each inserted record into totals.
|
|
||||||
const totalsAfterInsertSQL = `
|
|
||||||
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
|
|
||||||
UPDATE totals SET records = records + 1, size = size + NEW.size;
|
|
||||||
END
|
|
||||||
`
|
|
||||||
|
|
||||||
// totalsAfterDeleteSQL takes each deleted record out of totals.
|
|
||||||
const totalsAfterDeleteSQL = `
|
|
||||||
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
|
|
||||||
UPDATE totals SET records = records - 1, size = size - OLD.size;
|
|
||||||
END
|
|
||||||
`
|
|
||||||
|
|
||||||
// totalsAfterUpdateSQL replaces a record's old size in totals with its
|
|
||||||
// new one. scan's upsert of an existing record is such an update.
|
|
||||||
const totalsAfterUpdateSQL = `
|
|
||||||
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
|
|
||||||
UPDATE totals SET size = size + NEW.size - OLD.size;
|
|
||||||
END
|
|
||||||
`
|
|
||||||
|
|
||||||
// recordCountSQL reads the number of records from totals.
|
|
||||||
const recordCountSQL = `
|
|
||||||
SELECT records FROM totals
|
|
||||||
`
|
|
||||||
|
|
||||||
// upsertSQL inserts one file record, replacing any existing record for
|
// upsertSQL inserts one file record, replacing any existing record for
|
||||||
// the same path.
|
// the same path.
|
||||||
const upsertSQL = `
|
const upsertSQL = `
|
||||||
INSERT INTO files (
|
INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content)
|
||||||
path, size, disk_size, mtime, mtime_nsec, head, tail, content
|
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||||
)
|
|
||||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
|
||||||
ON CONFLICT (path) DO UPDATE SET
|
ON CONFLICT (path) DO UPDATE SET
|
||||||
size = excluded.size, disk_size = excluded.disk_size,
|
size = excluded.size, mtime = excluded.mtime,
|
||||||
mtime = excluded.mtime, mtime_nsec = excluded.mtime_nsec,
|
mtime_nsec = excluded.mtime_nsec,
|
||||||
head = excluded.head, tail = excluded.tail,
|
head = excluded.head, tail = excluded.tail,
|
||||||
content = excluded.content
|
content = excluded.content
|
||||||
`
|
`
|
||||||
@@ -146,44 +100,33 @@ var errHashSize = errors.New("hash is not 32 bytes")
|
|||||||
|
|
||||||
// databasePath resolves the database location: SFDUPES_DATABASE when
|
// databasePath resolves the database location: SFDUPES_DATABASE when
|
||||||
// set and non-empty, the compiled-in default otherwise. It returns the
|
// set and non-empty, the compiled-in default otherwise. It returns the
|
||||||
// path, made absolute by absolutePath, and where the path came from,
|
// path, with the working directory, its symlinks resolved, put in front
|
||||||
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
// of it when it is relative, and where the path came from, "default" or
|
||||||
// prints first on stderr.
|
// "from SFDUPES_DATABASE", for the line each subcommand prints first on
|
||||||
|
// stderr.
|
||||||
func databasePath() (string, string, error) {
|
func databasePath() (string, string, error) {
|
||||||
path, source := defaultDatabasePath, "default"
|
path, source := defaultDatabasePath, "default"
|
||||||
if p := os.Getenv(databaseEnv); p != "" {
|
if p := os.Getenv(databaseEnv); p != "" {
|
||||||
path, source = p, "from "+databaseEnv
|
path, source = p, "from "+databaseEnv
|
||||||
}
|
}
|
||||||
|
|
||||||
abs, err := absolutePath(path)
|
if filepath.IsAbs(path) {
|
||||||
|
return path, source, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// The working directory as the kernel reports it, with no symlink
|
||||||
|
// in it, as README "Database" says. os.Getwd may return PWD
|
||||||
|
// instead, which can name it through a symlink.
|
||||||
|
wd, err := unix.Getwd()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return abs, source, nil
|
// Not filepath.Abs or filepath.Join: they clean the path, which
|
||||||
}
|
// drops each ".." together with the directory name before it. When
|
||||||
|
// that directory is a symlink, the cleaned path names another file.
|
||||||
// absolutePath returns path unchanged when it is absolute, and
|
|
||||||
// otherwise the working directory, a "/" and path. The working
|
|
||||||
// directory is the one the kernel reports, with no symlink in it;
|
|
||||||
// os.Getwd may return PWD instead, which can name it through a symlink.
|
|
||||||
// The result is not cleaned, as filepath.Abs and filepath.Join would
|
|
||||||
// clean it. Cleaning drops each ".." together with the directory name
|
|
||||||
// before it. When that name is a symlink, the cleaned path names
|
|
||||||
// another file than the one the system reaches by following the symlink
|
|
||||||
// first.
|
|
||||||
func absolutePath(path string) (string, error) {
|
|
||||||
if filepath.IsAbs(path) {
|
|
||||||
return path, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
wd, err := unix.Getwd()
|
|
||||||
if err != nil {
|
|
||||||
return "", err
|
|
||||||
}
|
|
||||||
|
|
||||||
// The working directory ends in "/" only when it is the root.
|
// The working directory ends in "/" only when it is the root.
|
||||||
return strings.TrimSuffix(wd, "/") + "/" + path, nil
|
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// createDatabaseDirectory creates the directory the database at path
|
// createDatabaseDirectory creates the directory the database at path
|
||||||
@@ -207,14 +150,9 @@ func createDatabaseDirectory(path string) error {
|
|||||||
// scanParams are the connection parameters for scan: read-write, with
|
// scanParams are the connection parameters for scan: read-write, with
|
||||||
// WAL journaling and a busy timeout, so a report can run while a cron
|
// WAL journaling and a busy timeout, so a report can run while a cron
|
||||||
// scan is in progress. closeScanDatabase leaves WAL mode again.
|
// scan is in progress. closeScanDatabase leaves WAL mode again.
|
||||||
//
|
|
||||||
// Each write to files fires a trigger, so SQLite keeps a statement
|
|
||||||
// journal for it. temp_store(MEMORY) keeps that journal in memory,
|
|
||||||
// which costs far less than a temporary file.
|
|
||||||
const scanParams = "_pragma=busy_timeout(10000)" +
|
const scanParams = "_pragma=busy_timeout(10000)" +
|
||||||
"&_pragma=journal_mode(WAL)" +
|
"&_pragma=journal_mode(WAL)" +
|
||||||
"&_pragma=synchronous(NORMAL)" +
|
"&_pragma=synchronous(NORMAL)"
|
||||||
"&_pragma=temp_store(MEMORY)"
|
|
||||||
|
|
||||||
// reportParams are the connection parameters for report, trees and
|
// reportParams are the connection parameters for report, trees and
|
||||||
// info: read-only, with the same busy timeout. They set no journal
|
// info: read-only, with the same busy timeout. They set no journal
|
||||||
@@ -488,14 +426,14 @@ func createSchema(ctx context.Context, db *sql.DB) error {
|
|||||||
|
|
||||||
defer func() { _ = tx.Rollback() }()
|
defer func() { _ = tx.Rollback() }()
|
||||||
|
|
||||||
for _, stmt := range []string{
|
_, err = tx.ExecContext(ctx, createTableSQL)
|
||||||
createTableSQL, createIndexSQL, createTotalsSQL, insertTotalsSQL,
|
|
||||||
totalsAfterInsertSQL, totalsAfterDeleteSQL, totalsAfterUpdateSQL,
|
|
||||||
} {
|
|
||||||
_, err = tx.ExecContext(ctx, stmt)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("create schema: %w", err)
|
return fmt.Errorf("create schema: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
_, err = tx.ExecContext(ctx, createIndexSQL)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("create schema: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = tx.ExecContext(ctx,
|
_, err = tx.ExecContext(ctx,
|
||||||
@@ -542,7 +480,7 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
|||||||
|
|
||||||
var n int64
|
var n int64
|
||||||
|
|
||||||
err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("count records: %w", err)
|
return fmt.Errorf("count records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -550,8 +488,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
|||||||
total(n)
|
total(n)
|
||||||
|
|
||||||
rows, err := tx.QueryContext(ctx,
|
rows, err := tx.QueryContext(ctx,
|
||||||
"SELECT path, size, disk_size, mtime, mtime_nsec, head, tail, "+
|
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
|
||||||
"content FROM files ORDER BY path")
|
"FROM files ORDER BY path")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("read records: %w", err)
|
return fmt.Errorf("read records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -565,8 +503,7 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
|||||||
r scanRec
|
r scanRec
|
||||||
)
|
)
|
||||||
|
|
||||||
err = rows.Scan(&path, &r.size, &r.diskSize, &sec, &nsec,
|
err = rows.Scan(&path, &r.size, &sec, &nsec, &head, &tail, &content)
|
||||||
&head, &tail, &content)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("read record: %w", err)
|
return fmt.Errorf("read record: %w", err)
|
||||||
}
|
}
|
||||||
@@ -601,11 +538,10 @@ ORDER BY size DESC, head DESC, tail DESC, content DESC
|
|||||||
`
|
`
|
||||||
|
|
||||||
// loadSignatureRows gives total the number of records, then streams
|
// loadSignatureRows gives total the number of records, then streams
|
||||||
// every record's signature and path to fn, without its mtime or its
|
// every record's signature and path to fn, without its mtime, in the
|
||||||
// disk_size, in the order of signatureRowsSQL. The count and the
|
// order of signatureRowsSQL. The count and the records are read in one
|
||||||
// records are read in one transaction, so they agree while a scan is
|
// transaction, so they agree while a scan is committing. An error from
|
||||||
// committing. An error from fn stops the reading and is returned as it
|
// fn stops the reading and is returned as it is.
|
||||||
// is.
|
|
||||||
func loadSignatureRows(ctx context.Context, db *sql.DB,
|
func loadSignatureRows(ctx context.Context, db *sql.DB,
|
||||||
total func(n int64), fn func(r scanRec) error,
|
total func(n int64), fn func(r scanRec) error,
|
||||||
) error {
|
) error {
|
||||||
@@ -620,7 +556,7 @@ func loadSignatureRows(ctx context.Context, db *sql.DB,
|
|||||||
|
|
||||||
var n int64
|
var n int64
|
||||||
|
|
||||||
err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("count records: %w", err)
|
return fmt.Errorf("count records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -686,7 +622,7 @@ func (r *scanRec) setHashes(head, tail, content []byte) error {
|
|||||||
func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
||||||
var n int64
|
var n int64
|
||||||
|
|
||||||
err := db.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("count records: %w", err)
|
return 0, fmt.Errorf("count records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -694,10 +630,12 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
|||||||
return n, nil
|
return n, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// databaseCountsSQL reads the number of records and the sizes of all
|
// databaseCountsSQL counts the records and adds up their sizes, in one
|
||||||
// records added up from totals.
|
// pass over the table. A SUM over no rows is NULL, so on an empty table
|
||||||
|
// the SUM falls back to 0.
|
||||||
const databaseCountsSQL = `
|
const databaseCountsSQL = `
|
||||||
SELECT records, size FROM totals
|
SELECT COUNT(*), COALESCE(SUM(size), 0)
|
||||||
|
FROM files
|
||||||
`
|
`
|
||||||
|
|
||||||
// databaseCounts is what info reads from the database: its schema
|
// databaseCounts is what info reads from the database: its schema
|
||||||
@@ -740,15 +678,15 @@ func readDatabaseCounts(ctx context.Context,
|
|||||||
return c, nil
|
return c, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// loadFileMeta streams every record's path, size, disk_size and mtime
|
// loadFileMeta streams every record's path, size and mtime to fn. Scan
|
||||||
// to fn. Scan change detection needs no hash values, and skipping the
|
// change detection needs no hash values, and skipping the hash columns
|
||||||
// hash columns keeps the scan's in-memory index small on
|
// keeps the scan's in-memory index small on multi-million-file
|
||||||
// multi-million-file databases.
|
// databases.
|
||||||
func loadFileMeta(ctx context.Context, db *sql.DB,
|
func loadFileMeta(ctx context.Context, db *sql.DB,
|
||||||
fn func(path string, m fileMeta),
|
fn func(path string, size int64, mtime time.Time),
|
||||||
) error {
|
) error {
|
||||||
rows, err := db.QueryContext(ctx,
|
rows, err := db.QueryContext(ctx,
|
||||||
"SELECT path, size, disk_size, mtime, mtime_nsec FROM files")
|
"SELECT path, size, mtime, mtime_nsec FROM files")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("read records: %w", err)
|
return fmt.Errorf("read records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -758,18 +696,15 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
|
|||||||
for rows.Next() {
|
for rows.Next() {
|
||||||
var (
|
var (
|
||||||
path []byte
|
path []byte
|
||||||
m fileMeta
|
size, sec, nsec int64
|
||||||
sec, nsec int64
|
|
||||||
)
|
)
|
||||||
|
|
||||||
err = rows.Scan(&path, &m.size, &m.diskSize, &sec, &nsec)
|
err = rows.Scan(&path, &size, &sec, &nsec)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("read record: %w", err)
|
return fmt.Errorf("read record: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
m.mtime = time.Unix(sec, nsec)
|
fn(string(path), size, time.Unix(sec, nsec))
|
||||||
|
|
||||||
fn(string(path), m)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
err = rows.Err()
|
err = rows.Err()
|
||||||
@@ -787,21 +722,21 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
|
|||||||
// WAL small and let concurrent reports observe progress.
|
// WAL small and let concurrent reports observe progress.
|
||||||
const updateBatchSize = 10000
|
const updateBatchSize = 10000
|
||||||
|
|
||||||
// applyChanges writes upserts for new and changed files and deletes for
|
// applyChanges writes one scan's database changes — upserts for new and
|
||||||
// vanished ones, in batched transactions. Progress is rendered on prog
|
// changed files, deletes for vanished ones — in batched transactions.
|
||||||
// (one increment per change).
|
// Progress is rendered on prog (one increment per change).
|
||||||
func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
|
func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||||
deletes []string, prog *progress,
|
deletes []string, prog *progress,
|
||||||
) error {
|
) error {
|
||||||
for batch := range slices.Chunk(upserts, updateBatchSize) {
|
for batch := range slices.Chunk(upserts, updateBatchSize) {
|
||||||
err := applyBatch(ctx, db, batch, nil, nil, prog)
|
err := applyBatch(ctx, db, batch, nil, prog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
for batch := range slices.Chunk(deletes, updateBatchSize) {
|
for batch := range slices.Chunk(deletes, updateBatchSize) {
|
||||||
err := applyBatch(ctx, db, nil, nil, batch, prog)
|
err := applyBatch(ctx, db, nil, batch, prog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -810,26 +745,10 @@ func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// applyDiskSizes writes the disk_size of each of recs into its record,
|
// applyBatch commits one batch of upserts and deletes in a single
|
||||||
// in batched transactions, and leaves the rest of the record alone.
|
// transaction.
|
||||||
// Progress is rendered on prog (one increment per record).
|
|
||||||
func applyDiskSizes(ctx context.Context, db *sql.DB, recs []fileRec,
|
|
||||||
prog *progress,
|
|
||||||
) error {
|
|
||||||
for batch := range slices.Chunk(recs, updateBatchSize) {
|
|
||||||
err := applyBatch(ctx, db, nil, batch, nil, prog)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// applyBatch commits one batch of upserts, disk_size updates and
|
|
||||||
// deletes in a single transaction.
|
|
||||||
func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
|
func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||||
diskSizes []fileRec, deletes []string, prog *progress,
|
deletes []string, prog *progress,
|
||||||
) error {
|
) error {
|
||||||
tx, err := db.BeginTx(ctx, nil)
|
tx, err := db.BeginTx(ctx, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -843,11 +762,6 @@ func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
err = execDiskSizes(ctx, tx, diskSizes, prog)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = execDeletes(ctx, tx, deletes, prog)
|
err = execDeletes(ctx, tx, deletes, prog)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -873,7 +787,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
|
|||||||
defer func() { _ = st.Close() }()
|
defer func() { _ = st.Close() }()
|
||||||
|
|
||||||
for _, r := range upserts {
|
for _, r := range upserts {
|
||||||
_, err = st.ExecContext(ctx, []byte(r.path), r.size, r.diskSize,
|
_, err = st.ExecContext(ctx, []byte(r.path), r.size,
|
||||||
r.mtime.Unix(), r.mtime.Nanosecond(),
|
r.mtime.Unix(), r.mtime.Nanosecond(),
|
||||||
r.head[:], r.tail[:], r.content[:])
|
r.head[:], r.tail[:], r.content[:])
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -886,31 +800,6 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// execDiskSizes sets the disk_size of the record of each of recs. It
|
|
||||||
// does not touch size, so the totals triggers do not fire.
|
|
||||||
func execDiskSizes(ctx context.Context, tx *sql.Tx, recs []fileRec,
|
|
||||||
prog *progress,
|
|
||||||
) error {
|
|
||||||
st, err := tx.PrepareContext(ctx,
|
|
||||||
"UPDATE files SET disk_size = ? WHERE path = ?")
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("prepare disk_size update: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = st.Close() }()
|
|
||||||
|
|
||||||
for _, r := range recs {
|
|
||||||
_, err = st.ExecContext(ctx, r.diskSize, []byte(r.path))
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("update disk_size of %s: %w", r.path, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
prog.increment()
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// execDeletes removes the records for paths no longer present.
|
// execDeletes removes the records for paths no longer present.
|
||||||
func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
|
func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
|
||||||
prog *progress,
|
prog *progress,
|
||||||
|
|||||||
+10
-157
@@ -535,12 +535,17 @@ func TestLoadRowsRefuseWrongSizeHash(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// queryPlan returns the detail of each line of SQLite's EXPLAIN QUERY
|
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
|
||||||
// PLAN for query.
|
// from the files_signature index alone, with nothing to sort: the
|
||||||
func queryPlan(t *testing.T, db *sql.DB, query string) []string {
|
// first record comes at once, so report's read phase counts each
|
||||||
t.Helper()
|
// record as it comes rather than waiting on SQLite for all of them.
|
||||||
|
func TestSignatureRowsNeedNoSort(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
rows, err := db.QueryContext(t.Context(), "EXPLAIN QUERY PLAN "+query)
|
db := openTestDB(t)
|
||||||
|
|
||||||
|
rows, err := db.QueryContext(t.Context(),
|
||||||
|
"EXPLAIN QUERY PLAN "+signatureRowsSQL)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -568,164 +573,12 @@ func queryPlan(t *testing.T, db *sql.DB, query string) []string {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return plan
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
|
|
||||||
// from the files_signature index alone, with nothing to sort: the
|
|
||||||
// first record comes at once, so report's read phase counts each
|
|
||||||
// record as it comes rather than waiting on SQLite for all of them.
|
|
||||||
func TestSignatureRowsNeedNoSort(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
plan := queryPlan(t, db, signatureRowsSQL)
|
|
||||||
|
|
||||||
want := []string{"SCAN files USING COVERING INDEX files_signature"}
|
want := []string{"SCAN files USING COVERING INDEX files_signature"}
|
||||||
if !slices.Equal(plan, want) {
|
if !slices.Equal(plan, want) {
|
||||||
t.Errorf("query plan = %q, want %q", plan, want)
|
t.Errorf("query plan = %q, want %q", plan, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestCountsReadOnlyTotals checks that SQLite answers the queries for
|
|
||||||
// the number of records and their total size from the one row of
|
|
||||||
// totals. A plan that reads files or its index reads every record.
|
|
||||||
func TestCountsReadOnlyTotals(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
for _, query := range []string{recordCountSQL, databaseCountsSQL} {
|
|
||||||
plan := queryPlan(t, db, query)
|
|
||||||
|
|
||||||
want := []string{"SCAN totals"}
|
|
||||||
if !slices.Equal(plan, want) {
|
|
||||||
t.Errorf("query plan of %q = %q, want %q", query, plan, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCountsComeFromTotals sets the row of totals to numbers that files
|
|
||||||
// does not hold, then checks that scan's load phase, the read phases of
|
|
||||||
// trees and report, and info each report the numbers from totals.
|
|
||||||
func TestCountsComeFromTotals(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// files is empty, so a count over files gives 0 records of 0 bytes.
|
|
||||||
const records, size = 7, 9000
|
|
||||||
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
_, err := db.ExecContext(t.Context(),
|
|
||||||
"UPDATE totals SET records = ?, size = ?", records, size)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
scanTotal, err := countRecords(t.Context(), db)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var treesTotal, reportTotal int64
|
|
||||||
|
|
||||||
err = loadFileRows(t.Context(), db,
|
|
||||||
func(n int64) { treesTotal = n }, func(scanRec) {})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = loadSignatureRows(t.Context(), db,
|
|
||||||
func(n int64) { reportTotal = n },
|
|
||||||
func(scanRec) error { return nil })
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
info, err := readDatabaseCounts(t.Context(), db)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, got := range map[string]int64{
|
|
||||||
"scan's load": scanTotal,
|
|
||||||
"trees' read": treesTotal,
|
|
||||||
"report's read": reportTotal,
|
|
||||||
"info": info.records,
|
|
||||||
} {
|
|
||||||
if got != records {
|
|
||||||
t.Errorf("%s: %d records, want %d from totals",
|
|
||||||
name, got, records)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if info.size != size {
|
|
||||||
t.Errorf("info: size %d, want %d from totals", info.size, size)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// assertTotalsMatchFiles fails unless the row of totals holds the
|
|
||||||
// number of records and the sizes of all records added up, as counted
|
|
||||||
// over files.
|
|
||||||
func assertTotalsMatchFiles(t *testing.T, db *sql.DB) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
got, err := readDatabaseCounts(t.Context(), db)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var records, size int64
|
|
||||||
|
|
||||||
err = db.QueryRowContext(t.Context(),
|
|
||||||
"SELECT COUNT(*), COALESCE(SUM(size), 0) FROM files").
|
|
||||||
Scan(&records, &size)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got.records != records || got.size != size {
|
|
||||||
t.Errorf("totals = %d records of %d bytes, files holds %d of %d",
|
|
||||||
got.records, got.size, records, size)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestTotalsMatchFiles scans the smoke tree, rescans it after two files
|
|
||||||
// change size, and rescans it after two files are removed. After each
|
|
||||||
// scan the row of totals must agree with files.
|
|
||||||
func TestTotalsMatchFiles(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := buildSmokeTree(t)
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
syncTree(t, db, dir)
|
|
||||||
assertTotalsMatchFiles(t, db)
|
|
||||||
|
|
||||||
writeFile(t, dir, "a/one.bin", pattern(10, 2500))
|
|
||||||
writeFile(t, dir, "t1/f1", pattern(30, 1000))
|
|
||||||
|
|
||||||
if st := syncTree(t, db, dir); st.updated != 2 {
|
|
||||||
t.Fatalf("rescan stats = %+v, want 2 updated", st)
|
|
||||||
}
|
|
||||||
|
|
||||||
assertTotalsMatchFiles(t, db)
|
|
||||||
|
|
||||||
for _, name := range []string{"a/unique.bin", "t2/sub/f2"} {
|
|
||||||
err := os.Remove(filepath.Join(dir, name))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if st := syncTree(t, db, dir); st.removed != 2 {
|
|
||||||
t.Fatalf("rescan stats = %+v, want 2 removed", st)
|
|
||||||
}
|
|
||||||
|
|
||||||
assertTotalsMatchFiles(t, db)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestApplyChangesRoundTrip(t *testing.T) {
|
func TestApplyChangesRoundTrip(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -1,37 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"context"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"os"
|
|
||||||
"runtime"
|
|
||||||
)
|
|
||||||
|
|
||||||
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
|
|
||||||
// root" runs with no flags, then what "sfdupes trees" runs, and ends
|
|
||||||
// with a line on stderr saying how to print the report again. The dwim
|
|
||||||
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
|
|
||||||
// does, but only during the scan. An error from the scan, an interrupt
|
|
||||||
// included, ends it before the report. During the report, a signal
|
|
||||||
// ends it at once, as it ends trees.
|
|
||||||
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
|
||||||
scanCtx, stop := interruptContext(ctx)
|
|
||||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
|
|
||||||
|
|
||||||
stop()
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
err = runTrees(ctx, stdout)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
fmt.Fprintln(os.Stderr,
|
|
||||||
`dwim: run "sfdupes trees" to print this report again`)
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
@@ -1,83 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"path/filepath"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
|
|
||||||
// dwim scans its root, then prints the trees report, the larger of
|
|
||||||
// two duplicate-tree pairs first, and ends by saying how to print
|
|
||||||
// the report again.
|
|
||||||
t.Setenv(databaseEnv, testDBPath(t))
|
|
||||||
|
|
||||||
root := realTempDir(t)
|
|
||||||
|
|
||||||
small := []string{
|
|
||||||
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
|
|
||||||
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
|
|
||||||
}
|
|
||||||
large := []string{
|
|
||||||
filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))),
|
|
||||||
filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))),
|
|
||||||
}
|
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
|
||||||
|
|
||||||
stderr := captureStderr(t)
|
|
||||||
|
|
||||||
err := runDwim(t.Context(), root, &stdout)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
|
||||||
}
|
|
||||||
|
|
||||||
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
|
|
||||||
if len(lines) != 3 {
|
|
||||||
t.Fatalf("stdout = %q, want a header and two groups",
|
|
||||||
stdout.String())
|
|
||||||
}
|
|
||||||
|
|
||||||
wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t"
|
|
||||||
if !strings.HasPrefix(lines[1], wantFirst) {
|
|
||||||
t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst)
|
|
||||||
}
|
|
||||||
|
|
||||||
wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t"
|
|
||||||
if !strings.HasPrefix(lines[2], wantSecond) {
|
|
||||||
t.Errorf("second group = %q, want it to start %q",
|
|
||||||
lines[2], wantSecond)
|
|
||||||
}
|
|
||||||
|
|
||||||
last := "\ndwim: run \"sfdupes trees\" to print this report again\n"
|
|
||||||
if got := stderr(); !strings.HasSuffix(got, last) {
|
|
||||||
t.Errorf("stderr = %q, want it to end %q", got, last)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
|
||||||
// A fatal scan error ends dwim before the report: a root that does
|
|
||||||
// not exist fails the scan as it fails scan itself.
|
|
||||||
t.Setenv(databaseEnv, testDBPath(t))
|
|
||||||
|
|
||||||
missing := filepath.Join(realTempDir(t), "missing")
|
|
||||||
|
|
||||||
var stdout bytes.Buffer
|
|
||||||
|
|
||||||
stderr := captureStderr(t)
|
|
||||||
|
|
||||||
err := runDwim(t.Context(), missing, &stdout)
|
|
||||||
if err == nil {
|
|
||||||
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := stdout.String(); got != "" {
|
|
||||||
t.Errorf("stdout = %q, want no report", got)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := stderr(); strings.Contains(got, "trees:") ||
|
|
||||||
strings.Contains(got, "dwim:") {
|
|
||||||
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
+2
-4
@@ -17,9 +17,7 @@ import (
|
|||||||
func TestDefaultExcludes(t *testing.T) {
|
func TestDefaultExcludes(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// README "Rules for the walk": the Linux and macOS lists, and none
|
// README "Rules for the walk": each system's list.
|
||||||
// on any other system. The Windows list exists in the code only: the
|
|
||||||
// binary does not build for Windows, so the README leaves it out.
|
|
||||||
cases := map[string][]string{
|
cases := map[string][]string{
|
||||||
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
|
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
|
||||||
goosDarwin: {
|
goosDarwin: {
|
||||||
@@ -286,7 +284,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
|
|||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := realTempDir(t)
|
dir := t.TempDir()
|
||||||
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
||||||
t.Skip(dir + " is not inside a default exclude")
|
t.Skip(dir + " is not inside a default exclude")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -26,11 +26,11 @@ func infoPhase(name string) string {
|
|||||||
// it. Any database problem, including a missing database, is fatal.
|
// it. Any database problem, including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in two phases, each under a progress display of its
|
// The work runs in two phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database) and count (read the number of
|
// own on stderr: open (open the database) and count (count the records,
|
||||||
// records and their total size from the totals table). Each display
|
// with one query). Each display ends before the next phase starts, and
|
||||||
// ends before the next phase starts, and before an error is returned
|
// before an error is returned to be printed. The details are written
|
||||||
// to be printed. The details are written once the last display has
|
// once the last display has ended, so no display is drawn while they
|
||||||
// ended, so no display is drawn while they are written.
|
// are written.
|
||||||
func runInfo(ctx context.Context, stdout io.Writer) error {
|
func runInfo(ctx context.Context, stdout io.Writer) error {
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -56,7 +56,7 @@ func runInfo(ctx context.Context, stdout io.Writer) error {
|
|||||||
|
|
||||||
defer func() { _ = db.Close() }()
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
// One row holds both counts, so the phase never has a total.
|
// One query counts everything, so the phase never has a total.
|
||||||
prog = newProgress(progressParams{
|
prog = newProgress(progressParams{
|
||||||
label: infoPhase("count"), total: -1,
|
label: infoPhase("count"), total: -1,
|
||||||
waitingFor: countingRecords,
|
waitingFor: countingRecords,
|
||||||
|
|||||||
@@ -14,7 +14,6 @@
|
|||||||
// sfdupes report > dupes.tsv
|
// sfdupes report > dupes.tsv
|
||||||
// sfdupes trees > dupetrees.tsv
|
// sfdupes trees > dupetrees.tsv
|
||||||
// sfdupes info
|
// sfdupes info
|
||||||
// sfdupes dwim
|
|
||||||
// sfdupes --version
|
// sfdupes --version
|
||||||
//
|
//
|
||||||
// See README.md for the complete specification.
|
// See README.md for the complete specification.
|
||||||
@@ -46,7 +45,6 @@ const (
|
|||||||
cmdReport = "report"
|
cmdReport = "report"
|
||||||
cmdTrees = "trees"
|
cmdTrees = "trees"
|
||||||
cmdInfo = "info"
|
cmdInfo = "info"
|
||||||
cmdDwim = "dwim"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// errNoSubcommand is returned by the root command when it is invoked
|
// errNoSubcommand is returned by the root command when it is invoked
|
||||||
@@ -98,8 +96,8 @@ func main() {
|
|||||||
// exit code. It is the program's single exit point: the subcommands
|
// exit code. It is the program's single exit point: the subcommands
|
||||||
// return their errors instead of exiting, so every deferred cleanup —
|
// return their errors instead of exiting, so every deferred cleanup —
|
||||||
// above all closing the database, which checkpoints the SQLite WAL —
|
// above all closing the database, which checkpoints the SQLite WAL —
|
||||||
// runs before the process ends. The report, trees, info and dwim
|
// runs before the process ends. The report, trees and info subcommands
|
||||||
// subcommands write their data to stdout.
|
// write their data to stdout.
|
||||||
func run(args []string, stdout, stderr io.Writer) int {
|
func run(args []string, stdout, stderr io.Writer) int {
|
||||||
// A nil slice makes cobra fall back to os.Args, which would let a
|
// A nil slice makes cobra fall back to os.Args, which would let a
|
||||||
// test binary's own flags reach the command tree.
|
// test binary's own flags reach the command tree.
|
||||||
@@ -203,16 +201,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
|||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
|
|
||||||
dwimCmd := &cobra.Command{
|
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
|
||||||
Use: cmdDwim,
|
|
||||||
Short: "Scan / with the default excludes, then print the duplicate-tree report",
|
|
||||||
Args: cobra.NoArgs,
|
|
||||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
|
||||||
return runDwim(ctx, "/", stdout)
|
|
||||||
}),
|
|
||||||
}
|
|
||||||
|
|
||||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
|
|
||||||
|
|
||||||
return root
|
return root
|
||||||
}
|
}
|
||||||
|
|||||||
+60
-403
@@ -14,8 +14,6 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"golang.org/x/sys/unix"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// usageMarker is the first line of cobra's usage text, which a usage
|
// usageMarker is the first line of cobra's usage text, which a usage
|
||||||
@@ -182,9 +180,9 @@ func capture(t *testing.T, std **os.File) func() string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// brokenDatabase writes a database that opens cleanly and passes the
|
// brokenDatabase writes a database that opens cleanly and passes the
|
||||||
// schema-version check but has no tables, so the first query fails
|
// schema-version check but has no files table, so the first query
|
||||||
// with the database already open: a fatal error on a path that owns an
|
// fails with the database already open: a fatal error on a path that
|
||||||
// open database. It closes the database the way scan does.
|
// owns an open database. It closes the database the way scan does.
|
||||||
func brokenDatabase(t *testing.T) string {
|
func brokenDatabase(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -267,9 +265,8 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) {
|
|||||||
assertFatalOutput(t, stderr.String(), stdout())
|
assertFatalOutput(t, stderr.String(), stdout())
|
||||||
|
|
||||||
// Proof that the failure happened after the open: only a
|
// Proof that the failure happened after the open: only a
|
||||||
// query against the opened database can report this. Each
|
// query against the opened database can report this.
|
||||||
// subcommand's first query reads totals.
|
if !strings.Contains(stderr.String(), "no such table: files") {
|
||||||
if !strings.Contains(stderr.String(), "no such table: totals") {
|
|
||||||
t.Errorf("stderr = %q, want the failure to come from a "+
|
t.Errorf("stderr = %q, want the failure to come from a "+
|
||||||
"query on the open database", stderr.String())
|
"query on the open database", stderr.String())
|
||||||
}
|
}
|
||||||
@@ -396,7 +393,7 @@ func TestRunNamesRelativeDatabase(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
|
func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
|
||||||
// README §Settings: the path names the file exactly. With link a
|
// README §Database: the path names the file exactly. With link a
|
||||||
// symlink to the directory real/sub, link/../db.sqlite names
|
// symlink to the directory real/sub, link/../db.sqlite names
|
||||||
// real/db.sqlite, not db.sqlite beside link. Every subcommand must
|
// real/db.sqlite, not db.sqlite beside link. Every subcommand must
|
||||||
// use that file, relative or absolute, and its first line must name
|
// use that file, relative or absolute, and its first line must name
|
||||||
@@ -622,7 +619,6 @@ func TestRunUsageErrors(t *testing.T) {
|
|||||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
||||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
||||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||||
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
|
||||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||||
"unknown subcommand": {[]string{"nope"}, "unknown command"},
|
"unknown subcommand": {[]string{"nope"}, "unknown command"},
|
||||||
}
|
}
|
||||||
@@ -652,7 +648,7 @@ func TestRunUsageErrors(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
|
func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
|
||||||
// README §Settings: --workers below 1 is a usage error reported in
|
// README §scan mode: --workers below 1 is a usage error reported in
|
||||||
// one line on stderr, before the scan opens the database.
|
// one line on stderr, before the scan opens the database.
|
||||||
for _, workers := range []string{"0", "-1"} {
|
for _, workers := range []string{"0", "-1"} {
|
||||||
t.Run(workers, func(t *testing.T) {
|
t.Run(workers, func(t *testing.T) {
|
||||||
@@ -690,8 +686,8 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
|
|||||||
func TestRunHelp(t *testing.T) {
|
func TestRunHelp(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// README §Settings: help goes to stderr, exits 0, and leaves stdout
|
// README §Subcommands: help goes to stderr, exits 0, and leaves
|
||||||
// empty.
|
// stdout empty.
|
||||||
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
|
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
|
||||||
|
|
||||||
for _, args := range cases {
|
for _, args := range cases {
|
||||||
@@ -717,8 +713,8 @@ func TestRunHelp(t *testing.T) {
|
|||||||
func TestRunVersion(t *testing.T) {
|
func TestRunVersion(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// README §Settings: the version is one line on stdout, with nothing
|
// README §Subcommands: the version is one line on stdout, with
|
||||||
// on stderr, and exits 0.
|
// nothing on stderr, and exits 0.
|
||||||
info, _ := debug.ReadBuildInfo()
|
info, _ := debug.ReadBuildInfo()
|
||||||
|
|
||||||
for _, arg := range []string{"--version", "-v"} {
|
for _, arg := range []string{"--version", "-v"} {
|
||||||
@@ -777,8 +773,8 @@ func TestBuildVersion(t *testing.T) {
|
|||||||
func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// README §Error handling: a failed write to stdout is a fatal error,
|
// README §Error handling: a stdout write failure exits 1, reported
|
||||||
// exit 1, reported on stderr starting with "sfdupes:".
|
// in one line on stderr.
|
||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
|
|
||||||
code := run([]string{"--version"}, failingWriter{}, &stderr)
|
code := run([]string{"--version"}, failingWriter{}, &stderr)
|
||||||
@@ -800,7 +796,7 @@ func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
|||||||
func scanFixture(t *testing.T) []string {
|
func scanFixture(t *testing.T) []string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
dir := realTempDir(t)
|
dir := t.TempDir()
|
||||||
|
|
||||||
dupes := []string{
|
dupes := []string{
|
||||||
writeFile(t, dir, "one/a.bin", pattern(1, 300)),
|
writeFile(t, dir, "one/a.bin", pattern(1, 300)),
|
||||||
@@ -821,23 +817,6 @@ func scanFixture(t *testing.T) []string {
|
|||||||
return dupes
|
return dupes
|
||||||
}
|
}
|
||||||
|
|
||||||
// storedDiskSize returns the disk_size of the record for path in the
|
|
||||||
// database at dbPath. A test of a trees run compares with this rather
|
|
||||||
// than with a new lstat, which on ZFS can differ from the scan's (see
|
|
||||||
// diskSizeOf).
|
|
||||||
func storedDiskSize(t *testing.T, dbPath, path string) int64 {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
db, err := openReportDatabase(t.Context(), dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
return recordByPath(t, dbRecords(t, db), path).diskSize
|
|
||||||
}
|
|
||||||
|
|
||||||
// scanOK runs scan over operands, fails the test unless it exits 0 with
|
// scanOK runs scan over operands, fails the test unless it exits 0 with
|
||||||
// nothing on stdout, and returns everything it printed to stderr.
|
// nothing on stdout, and returns everything it printed to stderr.
|
||||||
func scanOK(t *testing.T, operands ...string) string {
|
func scanOK(t *testing.T, operands ...string) string {
|
||||||
@@ -870,7 +849,7 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
|||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := realTempDir(t)
|
dir := t.TempDir()
|
||||||
writeFile(t, dir, "a.bin", pattern(1, 300))
|
writeFile(t, dir, "a.bin", pattern(1, 300))
|
||||||
|
|
||||||
// Same size as a.bin, so the scan reads it, and the read fails.
|
// Same size as a.bin, so the scan reads it, and the read fails.
|
||||||
@@ -898,292 +877,75 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
|||||||
assertNoSidecars(t, path)
|
assertNoSidecars(t, path)
|
||||||
}
|
}
|
||||||
|
|
||||||
// realTempDir returns a new temporary directory by its real path, so
|
func TestRunScanSkipsSymlinkOperand(t *testing.T) {
|
||||||
// tests hold where the temporary directory is reached through a
|
|
||||||
// symlink.
|
|
||||||
func realTempDir(t *testing.T) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
dir, err := filepath.EvalSymlinks(t.TempDir())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return dir
|
|
||||||
}
|
|
||||||
|
|
||||||
// symlink creates link pointing at target.
|
|
||||||
func symlink(t *testing.T, target, link string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
err := os.Symlink(target, link)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
|
|
||||||
// The operand is scanned at its real path, and its file is recorded
|
|
||||||
// there.
|
|
||||||
func TestRunScanFollowsSymlinkOperand(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := realTempDir(t)
|
dir := t.TempDir()
|
||||||
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||||
|
|
||||||
link := filepath.Join(dir, "link")
|
link := filepath.Join(dir, "link")
|
||||||
symlink(t, filepath.Join(dir, "target"), link)
|
|
||||||
|
|
||||||
stderr := scanOK(t, link)
|
err := os.Symlink(filepath.Join(dir, "target"), link)
|
||||||
|
if err != nil {
|
||||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
t.Fatal(err)
|
||||||
"0 unchanged), 0 skipped\n"
|
|
||||||
if !strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
// Scanning a directory through the symlink stores a record beneath
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
// the symlink's own path for a file beneath its target.
|
||||||
}
|
scanOK(t, filepath.Join(link, "sub"))
|
||||||
|
|
||||||
|
assertOperandSkipped(t, path, link, "symlink",
|
||||||
|
filepath.Join(link, "sub", "f"))
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRunScanWalksOperandUnderSymlinkOperand scans link and link/sub,
|
|
||||||
// with link a symlink to target. Both resolve under target, so
|
|
||||||
// target/sub is walked once, as part of target.
|
|
||||||
func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
|
func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
|
||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := realTempDir(t)
|
dir := t.TempDir()
|
||||||
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||||
|
|
||||||
link := filepath.Join(dir, "link")
|
link := filepath.Join(dir, "link")
|
||||||
symlink(t, filepath.Join(dir, "target"), link)
|
|
||||||
|
|
||||||
stderr := scanOK(t, link, filepath.Join(link, "sub"))
|
err := os.Symlink(filepath.Join(dir, "target"), link)
|
||||||
|
|
||||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 0 skipped\n"
|
|
||||||
if !strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanWalksSymlinkedOperandBelowOperand is the case of
|
|
||||||
// https://git.eeqj.de/sneak/sfdupes/issues/144: scan dir dir/link/sub,
|
|
||||||
// with dir/link a symlink to a directory outside dir. The walk of dir
|
|
||||||
// does not follow link, so dir/link/sub is walked on its own at its
|
|
||||||
// real path, and the record an earlier scan stored there survives.
|
|
||||||
func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
a := writeFile(t, dir, "a", pattern(1, 10))
|
|
||||||
|
|
||||||
outside := realTempDir(t)
|
|
||||||
f := writeFile(t, outside, "sub/f", pattern(2, 10))
|
|
||||||
|
|
||||||
symlink(t, outside, filepath.Join(dir, "link"))
|
|
||||||
|
|
||||||
scanOK(t, filepath.Join(outside, "sub"))
|
|
||||||
|
|
||||||
stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
|
|
||||||
|
|
||||||
summary := "scan: 2 files seen (1 added, 0 updated, 0 removed, " +
|
|
||||||
"1 unchanged), 0 skipped\n"
|
|
||||||
if !strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []string{a, f}
|
|
||||||
slices.Sort(want)
|
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, want) {
|
|
||||||
t.Errorf("records = %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanDropsSymlinkOperandInsideOperand scans dir and link, with
|
|
||||||
// link, outside dir, a symlink to dir/real. link resolves inside dir,
|
|
||||||
// so it is dropped as covered by dir, and its file is recorded once.
|
|
||||||
func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
f := writeFile(t, dir, "real/f", pattern(1, 10))
|
|
||||||
|
|
||||||
link := filepath.Join(realTempDir(t), "link")
|
|
||||||
symlink(t, filepath.Join(dir, "real"), link)
|
|
||||||
|
|
||||||
stderr := scanOK(t, dir, link)
|
|
||||||
|
|
||||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 0 skipped\n"
|
|
||||||
if !strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and
|
|
||||||
// a symlink whose target does not exist. The scan fails with exit 1 and
|
|
||||||
// an error naming the symlink as given, before it walks anything, so no
|
|
||||||
// database is created.
|
|
||||||
func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
writeFile(t, dir, "f", pattern(1, 10))
|
|
||||||
|
|
||||||
dangling := filepath.Join(realTempDir(t), "dangling")
|
|
||||||
symlink(t, filepath.Join(dir, "missing"), dangling)
|
|
||||||
|
|
||||||
stdout := captureStdout(t)
|
|
||||||
|
|
||||||
var stderr bytes.Buffer
|
|
||||||
|
|
||||||
code := run([]string{cmdScan, dir, dangling}, os.Stdout, &stderr)
|
|
||||||
if code != exitFatal {
|
|
||||||
t.Errorf("run(scan %s %s) = %d, want %d", dir, dangling, code,
|
|
||||||
exitFatal)
|
|
||||||
}
|
|
||||||
|
|
||||||
assertFatalOutput(t, stderr.String(), stdout())
|
|
||||||
|
|
||||||
want := "sfdupes: resolve " + dangling + ": "
|
|
||||||
if !strings.Contains(stderr.String(), want) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr.String(), want)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err := os.Stat(path)
|
|
||||||
if !errors.Is(err, fs.ErrNotExist) {
|
|
||||||
t.Errorf("database: stat error %v, want it missing", err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath runs scan .
|
|
||||||
// after a cd through home/proj, a symlink to data/proj. The file is
|
|
||||||
// recorded under data/proj, its real path.
|
|
||||||
func TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
|
|
||||||
|
|
||||||
err := os.Mkdir(filepath.Join(dir, "home"), 0o750)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
symlink(t, filepath.Join(dir, "data", "proj"),
|
// link is dropped as a symlink, but link/sub must still be scanned,
|
||||||
filepath.Join(dir, "home", "proj"))
|
// not dropped as lying under link.
|
||||||
|
scanOK(t, link, filepath.Join(link, "sub"))
|
||||||
|
|
||||||
// t.Chdir also sets PWD to the path through the symlink, as a
|
db, err := openDB(path, reportParams)
|
||||||
// shell does after cd.
|
|
||||||
t.Chdir(filepath.Join(dir, "home", "proj"))
|
|
||||||
|
|
||||||
scanOK(t, ".")
|
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanFollowsSymlinkBeforeDotDot scans a path with a ".." after
|
|
||||||
// home/proj, a symlink to data/proj: home/proj/.. from dir, and .. after
|
|
||||||
// a cd through home/proj. As with realpath, the ".." counts from
|
|
||||||
// data/proj, so data is scanned and home is not.
|
|
||||||
func TestRunScanFollowsSymlinkBeforeDotDot(t *testing.T) {
|
|
||||||
for _, tc := range []struct{ wd, operand string }{
|
|
||||||
{"", "home/proj/.."},
|
|
||||||
{"home/proj", ".."},
|
|
||||||
} {
|
|
||||||
t.Run(tc.operand, func(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
|
|
||||||
g := writeFile(t, dir, "data/g", pattern(2, 10))
|
|
||||||
writeFile(t, dir, "home/h", pattern(3, 10))
|
|
||||||
symlink(t, filepath.Join(dir, "data", "proj"),
|
|
||||||
filepath.Join(dir, "home", "proj"))
|
|
||||||
|
|
||||||
// t.Chdir also sets PWD to the path given, as a shell does
|
|
||||||
// after cd.
|
|
||||||
t.Chdir(filepath.Join(dir, tc.wd))
|
|
||||||
|
|
||||||
scanOK(t, tc.operand)
|
|
||||||
|
|
||||||
want := []string{f, g}
|
|
||||||
slices.Sort(want)
|
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, want) {
|
|
||||||
t.Errorf("records = %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanSkipsSymlinkToFIFOOperand scans link, a symlink to a FIFO.
|
|
||||||
// The scan warns about the FIFO by its real path, counts it as skipped,
|
|
||||||
// and exits 0.
|
|
||||||
func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
|
|
||||||
t.Setenv(databaseEnv, testDBPath(t))
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
fifo := filepath.Join(dir, "fifo")
|
|
||||||
|
|
||||||
err := unix.Mkfifo(fifo, 0o600)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
link := filepath.Join(dir, "link")
|
t.Cleanup(func() { _ = db.Close() })
|
||||||
symlink(t, fifo, link)
|
|
||||||
|
|
||||||
stderr := scanOK(t, link)
|
recordByPath(t, dbRecords(t, db), filepath.Join(link, "sub", "f"))
|
||||||
|
|
||||||
warning := "walk " + fifo + ": skipping FIFO operand\n"
|
|
||||||
if !strings.Contains(stderr, warning) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
|
||||||
}
|
|
||||||
|
|
||||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 1 skipped\n"
|
|
||||||
if !strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
||||||
zfs := filepath.Join(realTempDir(t), ".zfs")
|
path := testDBPath(t)
|
||||||
f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
|
|
||||||
|
|
||||||
// No scan records a file inside a .zfs directory, so the record
|
|
||||||
// beneath the operand is written straight into the database.
|
|
||||||
path := seedDatabase(t, []scanRec{{size: 10, path: f}})
|
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
zfs := filepath.Join(t.TempDir(), ".zfs")
|
||||||
|
snapshot := filepath.Join(zfs, "snapshot", "hourly")
|
||||||
|
f := writeFile(t, snapshot, "f", pattern(1, 10))
|
||||||
|
|
||||||
|
// An operand beneath a .zfs directory is walked, because it is not
|
||||||
|
// itself named .zfs.
|
||||||
|
scanOK(t, snapshot)
|
||||||
|
|
||||||
assertOperandSkipped(t, path, zfs, ".zfs directory", f)
|
assertOperandSkipped(t, path, zfs, ".zfs directory", f)
|
||||||
}
|
}
|
||||||
|
|
||||||
// assertOperandSkipped scans operand alone and checks that it is skipped
|
// assertOperandSkipped scans operand alone and checks that it is skipped
|
||||||
// as kind: a warning naming it, one skip in the summary, exit 0, and the
|
// as kind: a warning naming it, one skip in the summary, exit 0, and the
|
||||||
// record for kept, stored beneath operand before this scan, still in the
|
// record for kept, which an earlier scan stored beneath operand, still
|
||||||
// database at dbPath.
|
// in the database at dbPath.
|
||||||
func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
|
func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
|
||||||
kept string,
|
kept string,
|
||||||
) {
|
) {
|
||||||
@@ -1212,107 +974,6 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
|
|||||||
recordByPath(t, dbRecords(t, db), kept)
|
recordByPath(t, dbRecords(t, db), kept)
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs
|
|
||||||
// directory, first alone and then under another operand. Each time it
|
|
||||||
// is warned about and counted as skipped, and nothing in it is walked.
|
|
||||||
// The other operand is scanned as usual.
|
|
||||||
func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
|
|
||||||
writeFile(t, hourly, "f", pattern(1, 10))
|
|
||||||
f := writeFile(t, dir, "f", pattern(2, 10))
|
|
||||||
|
|
||||||
warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
|
|
||||||
|
|
||||||
for _, tc := range []struct {
|
|
||||||
operands []string
|
|
||||||
summary string
|
|
||||||
}{
|
|
||||||
{
|
|
||||||
[]string{hourly},
|
|
||||||
"scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 1 skipped\n",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
[]string{dir, hourly},
|
|
||||||
"scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 1 skipped\n",
|
|
||||||
},
|
|
||||||
} {
|
|
||||||
stderr := scanOK(t, tc.operands...)
|
|
||||||
if !strings.Contains(stderr, warning) ||
|
|
||||||
!strings.Contains(stderr, tc.summary) {
|
|
||||||
t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands,
|
|
||||||
stderr, warning, tc.summary)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
db, err := openDB(path, reportParams)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) {
|
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
|
|
||||||
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
|
|
||||||
// by its real path, counted as skipped, and not walked.
|
|
||||||
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
|
||||||
t.Setenv(databaseEnv, testDBPath(t))
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
|
|
||||||
|
|
||||||
snaps := filepath.Join(dir, "snaps")
|
|
||||||
symlink(t, filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
|
|
||||||
|
|
||||||
stderr := scanOK(t, filepath.Join(snaps, "daily"))
|
|
||||||
|
|
||||||
daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
|
|
||||||
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
|
|
||||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 1 skipped\n"
|
|
||||||
|
|
||||||
if !strings.Contains(stderr, warning) ||
|
|
||||||
!strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
|
|
||||||
// a symlink to pool/data. The operand is scanned as usual, and its file
|
|
||||||
// is recorded by its real path.
|
|
||||||
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := realTempDir(t)
|
|
||||||
f := writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
|
|
||||||
|
|
||||||
data := filepath.Join(dir, "data")
|
|
||||||
symlink(t, filepath.Join(dir, "pool", "data"), data)
|
|
||||||
|
|
||||||
stderr := scanOK(t, filepath.Join(data, "daily"))
|
|
||||||
|
|
||||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 0 skipped\n"
|
|
||||||
if !strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunReportSucceeds(t *testing.T) {
|
func TestRunReportSucceeds(t *testing.T) {
|
||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
@@ -1367,9 +1028,8 @@ func TestRunTreesSucceeds(t *testing.T) {
|
|||||||
|
|
||||||
// The two directories holding the duplicate pair are duplicate
|
// The two directories holding the duplicate pair are duplicate
|
||||||
// trees of each other.
|
// trees of each other.
|
||||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
want := "first\tdupe\tfiles\tsize\n" +
|
||||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
|
||||||
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n"
|
|
||||||
if got := stdout.String(); got != want {
|
if got := stdout.String(); got != want {
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
@@ -1403,10 +1063,9 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
|
|||||||
cases := map[string]string{
|
cases := map[string]string{
|
||||||
cmdReport: "first\tdupe\tsize\n" +
|
cmdReport: "first\tdupe\tsize\n" +
|
||||||
dupes[0] + "\t" + dupes[1] + "\t300\n",
|
dupes[0] + "\t" + dupes[1] + "\t300\n",
|
||||||
cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
cmdTrees: "first\tdupe\tfiles\tsize\n" +
|
||||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
|
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
|
||||||
"\t1\t300\t" +
|
"\t1\t300\n",
|
||||||
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n",
|
|
||||||
}
|
}
|
||||||
|
|
||||||
for name, want := range cases {
|
for name, want := range cases {
|
||||||
@@ -1441,11 +1100,8 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
|
|||||||
t.Errorf("report stdout = %q, want %q", got, want)
|
t.Errorf("report stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
diskSize := storedDiskSize(t, filepath.Join(dir, name), dupes[1])
|
want = "first\tdupe\tfiles\tsize\n" +
|
||||||
|
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
|
||||||
want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
|
||||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
|
||||||
strconv.FormatInt(diskSize, 10) + "\n"
|
|
||||||
if got := runStdout(t, cmdTrees); got != want {
|
if got := runStdout(t, cmdTrees); got != want {
|
||||||
t.Errorf("trees stdout = %q, want %q", got, want)
|
t.Errorf("trees stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
@@ -1466,7 +1122,7 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunDatabasePathUsedAsGiven(t *testing.T) {
|
func TestRunDatabasePathUsedAsGiven(t *testing.T) {
|
||||||
// README §Settings: the path names the database file exactly. In
|
// README §Database: the path names the database file exactly. In
|
||||||
// SQLite's connection string an unescaped ? or # would end the file
|
// SQLite's connection string an unescaped ? or # would end the file
|
||||||
// name and % would start an escape, and a path starting with //
|
// name and % would start an escape, and a path starting with //
|
||||||
// could be read as a host name. %25 is a valid escape, so unescaped
|
// could be read as a host name. %25 is a valid escape, so unescaped
|
||||||
@@ -1511,8 +1167,9 @@ func holdScanLock(t *testing.T, path string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunSecondScanFails(t *testing.T) {
|
func TestRunSecondScanFails(t *testing.T) {
|
||||||
// README §Database: a second scan that finds the lock held fails
|
// README §Database: while one scan holds the lock, a second scan
|
||||||
// at once, naming the lock file, without opening the database.
|
// fails at once, naming the lock file, without opening the
|
||||||
|
// database.
|
||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
@@ -1571,7 +1228,7 @@ func assertSecondScanFails(t *testing.T, path string) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
|
func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
|
||||||
// README §Database: a path through a symlink names the same lock
|
// README §Database: every path to one database names the same lock
|
||||||
// file, so while a scan holds it through one path, a scan through
|
// file, so while a scan holds it through one path, a scan through
|
||||||
// another fails, naming the lock file beside the database's real
|
// another fails, naming the lock file beside the database's real
|
||||||
// path.
|
// path.
|
||||||
@@ -1676,8 +1333,8 @@ func TestRunReportsDuringScan(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunStdoutClosedIsFatal(t *testing.T) {
|
func TestRunStdoutClosedIsFatal(t *testing.T) {
|
||||||
// README §Error handling: a failed write to stdout is a fatal error,
|
// README §Error handling: a stdout write failure exits 1, reported
|
||||||
// exit 1, reported on stderr starting with "sfdupes:".
|
// in one line on stderr.
|
||||||
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
|
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
|
||||||
t.Run(name, func(t *testing.T) {
|
t.Run(name, func(t *testing.T) {
|
||||||
t.Setenv(databaseEnv, testDBPath(t))
|
t.Setenv(databaseEnv, testDBPath(t))
|
||||||
|
|||||||
+1
-1
@@ -60,7 +60,7 @@ const fallbackWidth = 80
|
|||||||
const spinnerMargin = 5
|
const spinnerMargin = 5
|
||||||
|
|
||||||
// countingRecords is what the display of a phase that reads the records
|
// countingRecords is what the display of a phase that reads the records
|
||||||
// says until it has the number of records, its total.
|
// says while it counts them, before it has its total.
|
||||||
const countingRecords = "counting the records"
|
const countingRecords = "counting the records"
|
||||||
|
|
||||||
// openPhase is the name of the phase in which report, trees and info
|
// openPhase is the name of the phase in which report, trees and info
|
||||||
|
|||||||
@@ -21,11 +21,9 @@ const minGroupSize = 2
|
|||||||
|
|
||||||
// scanRec is one file record from the database. The signature (size,
|
// scanRec is one file record from the database. The signature (size,
|
||||||
// head, tail, content) is the duplicate key; mtime is informational
|
// head, tail, content) is the duplicate key; mtime is informational
|
||||||
// only and used by scan for change detection. diskSize is the bytes the
|
// only and used by scan for change detection.
|
||||||
// file takes on disk, which trees adds up.
|
|
||||||
type scanRec struct {
|
type scanRec struct {
|
||||||
size int64
|
size int64
|
||||||
diskSize int64
|
|
||||||
mtime time.Time
|
mtime time.Time
|
||||||
head [sha256.Size]byte
|
head [sha256.Size]byte
|
||||||
tail [sha256.Size]byte
|
tail [sha256.Size]byte
|
||||||
@@ -47,11 +45,11 @@ func reportPhase(name string) string {
|
|||||||
// problem, including a missing database, is fatal.
|
// problem, including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in two phases, each under a progress display of its
|
// The work runs in two phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database) and read (read the number of
|
// own on stderr: open (open the database) and read (count the records,
|
||||||
// records from the totals table, then read the records in signature
|
// then read them in signature order, writing each size's duplicate
|
||||||
// order, writing each size's duplicate groups to stdout once that size
|
// groups to stdout once that size is read; see readReport). Each
|
||||||
// is read; see readReport). Each display ends before the next phase
|
// display ends before the next phase starts, and before an error is
|
||||||
// starts, and before an error is returned to be printed.
|
// returned to be printed.
|
||||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
func runReport(ctx context.Context, stdout io.Writer) error {
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
+2
-2
@@ -38,8 +38,8 @@ func awkwardPairRecs() []scanRec {
|
|||||||
h, tl, c := testHash("h"), testHash("t"), testHash("c")
|
h, tl, c := testHash("h"), testHash("t"), testHash("c")
|
||||||
|
|
||||||
return []scanRec{
|
return []scanRec{
|
||||||
{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: "/d/A/f"},
|
{size: 5, head: h, tail: tl, content: c, path: "/d/A/f"},
|
||||||
{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
|
{size: 5, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -52,25 +52,22 @@ const workQueueDepth = 1024
|
|||||||
// has already printed its line, so run prints nothing more.
|
// has already printed its line, so run prints nothing more.
|
||||||
var errInterrupted = errors.New("scan interrupted")
|
var errInterrupted = errors.New("scan interrupted")
|
||||||
|
|
||||||
// fileRec carries one statted file between the scan phases. diskSize is
|
// fileRec carries one statted file between the scan phases. dev and
|
||||||
// the bytes the file takes on disk. dev and ino identify the underlying
|
// ino identify the underlying inode so hard-linked paths can share
|
||||||
// inode so hard-linked paths can share one read; both are zero when the
|
// one read; both are zero when the platform exposes no inode.
|
||||||
// platform exposes no inode.
|
|
||||||
type fileRec struct {
|
type fileRec struct {
|
||||||
path string
|
path string
|
||||||
size int64
|
size int64
|
||||||
diskSize int64
|
|
||||||
mtime time.Time
|
mtime time.Time
|
||||||
dev uint64
|
dev uint64
|
||||||
ino uint64
|
ino uint64
|
||||||
}
|
}
|
||||||
|
|
||||||
// fileMeta is the in-memory index entry for one existing database
|
// fileMeta is the in-memory index entry for one existing database
|
||||||
// record: what the walk compares the file with. Hashes stay on disk; at
|
// record: just enough for change detection. Hashes stay on disk; at
|
||||||
// tens of millions of records they would dominate the scan's memory.
|
// tens of millions of records they would dominate the scan's memory.
|
||||||
type fileMeta struct {
|
type fileMeta struct {
|
||||||
size int64
|
size int64
|
||||||
diskSize int64
|
|
||||||
mtime time.Time
|
mtime time.Time
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -209,30 +206,29 @@ func interrupted(walked int) error {
|
|||||||
return errInterrupted
|
return errInterrupted
|
||||||
}
|
}
|
||||||
|
|
||||||
// resolveRoots converts each PATH operand to its real path: absolute,
|
// resolveRoots converts each PATH operand to an absolute, lexically
|
||||||
// with every symlink in it followed. Database records are keyed by
|
// cleaned path (symlinks are not resolved) and verifies that it
|
||||||
// that path, so scan results depend neither on the working directory
|
// exists. Database records are keyed by absolute path, so scan results
|
||||||
// nor on the symlinks an operand was given through. An operand that
|
// must not depend on the working directory.
|
||||||
// cannot be resolved, because it or a symlink's target does not exist
|
|
||||||
// or cannot be read, is a fatal error before any scanning.
|
|
||||||
func resolveRoots(roots []string) ([]string, error) {
|
func resolveRoots(roots []string) ([]string, error) {
|
||||||
resolved := make([]string, 0, len(roots))
|
abs := make([]string, 0, len(roots))
|
||||||
|
|
||||||
for _, root := range roots {
|
for _, root := range roots {
|
||||||
a, err := absolutePath(root)
|
a, err := filepath.Abs(root)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
r, err := filepath.EvalSymlinks(a)
|
// A nonexistent operand is a fatal error before any scanning.
|
||||||
|
_, err = os.Lstat(a)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
resolved = append(resolved, r)
|
abs = append(abs, a)
|
||||||
}
|
}
|
||||||
|
|
||||||
return resolved, nil
|
return abs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// pruneRoots drops operands already covered by another operand:
|
// pruneRoots drops operands already covered by another operand:
|
||||||
@@ -346,7 +342,6 @@ type scanState struct {
|
|||||||
db *sql.DB
|
db *sql.DB
|
||||||
existing map[string]fileMeta
|
existing map[string]fileMeta
|
||||||
toHash []fileRec // new and changed files: must be read
|
toHash []fileRec // new and changed files: must be read
|
||||||
diskSizes []fileRec // unchanged files whose disk_size changed
|
|
||||||
batch []scanRec
|
batch []scanRec
|
||||||
st scanStats
|
st scanStats
|
||||||
}
|
}
|
||||||
@@ -388,8 +383,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
|||||||
) error {
|
) error {
|
||||||
// Types and filesystems are checked before pruning so that an
|
// Types and filesystems are checked before pruning so that an
|
||||||
// operand under a dropped one is still scanned, not dropped as lying
|
// operand under a dropped one is still scanned, not dropped as lying
|
||||||
// under it, and so that an operand inside a .zfs directory gets its
|
// under it.
|
||||||
// warning even when it lies under another operand.
|
|
||||||
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
|
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
|
||||||
deviceOfPath)
|
deviceOfPath)
|
||||||
|
|
||||||
@@ -417,17 +411,15 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
|||||||
}
|
}
|
||||||
|
|
||||||
// walkableRoots returns the operands the walk can start from: regular
|
// walkableRoots returns the operands the walk can start from: regular
|
||||||
// files, and directories not named .zfs, none of them inside a
|
// files, and directories not named .zfs. Every other operand is warned
|
||||||
// directory named .zfs. The operands are real paths from resolveRoots,
|
// about, counted as skipped, and dropped. A dropped operand is no
|
||||||
// so these checks apply to what a symlink operand points at. Every
|
// longer a root, so the records stored beneath it are treated as if it
|
||||||
// other operand is warned about, counted as skipped, and dropped. A
|
// had not been given: those under no other root count as outside the
|
||||||
// dropped operand is no longer a root, so the records stored beneath it
|
// roots and are not deleted as unverified. An operand on a filesystem
|
||||||
// are treated as if it had not been given: those under no other root
|
// in rules that is not persistent is dropped the same way, with a line
|
||||||
// count as outside the roots and are not deleted as unverified. An
|
// saying so that is not a warning, and is not counted as skipped. An
|
||||||
// operand on a filesystem in rules that is not persistent is dropped
|
// operand that fails lstat here is kept, and the walk warns about it
|
||||||
// the same way, with a line saying so that is not a warning, and is not
|
// unless it no longer exists.
|
||||||
// counted as skipped. An operand that fails lstat here is kept, and the
|
|
||||||
// walk warns about it unless it no longer exists.
|
|
||||||
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
||||||
kept := make([]string, 0, len(roots))
|
kept := make([]string, 0, len(roots))
|
||||||
|
|
||||||
@@ -462,8 +454,8 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
|||||||
// included, so its display counts against the number of records.
|
// included, so its display counts against the number of records.
|
||||||
func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||||
// Indexing tens of millions of records takes real time; without a
|
// Indexing tens of millions of records takes real time; without a
|
||||||
// display the scan looks hung before the walk begins. The number of
|
// display the scan looks hung before the walk begins. Counting them
|
||||||
// records, read first, gives the display its total.
|
// first gives the display its total.
|
||||||
prog := newProgress(progressParams{
|
prog := newProgress(progressParams{
|
||||||
label: scanPhase("load"), unit: scanUnit, total: -1,
|
label: scanPhase("load"), unit: scanUnit, total: -1,
|
||||||
waitingFor: countingRecords, stalledOn: "db",
|
waitingFor: countingRecords, stalledOn: "db",
|
||||||
@@ -479,24 +471,24 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
|||||||
|
|
||||||
s.existing = make(map[string]fileMeta)
|
s.existing = make(map[string]fileMeta)
|
||||||
|
|
||||||
return loadFileMeta(ctx, s.db, func(path string, m fileMeta) {
|
return loadFileMeta(ctx, s.db,
|
||||||
|
func(path string, size int64, mtime time.Time) {
|
||||||
prog.increment()
|
prog.increment()
|
||||||
|
|
||||||
if underAnyRoot(path, roots) {
|
if underAnyRoot(path, roots) {
|
||||||
s.existing[path] = m
|
s.existing[path] = fileMeta{size: size, mtime: mtime}
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
// walkPhase drains the walk and queues every new or changed file for
|
// walkPhase drains the walk and queues every new or changed file for
|
||||||
// the hash phase. An unchanged file is not read, and its record counts
|
// the hash phase. An unchanged file needs nothing more, and its record
|
||||||
// as verified. When its disk_size differs from the record's, it is
|
// counts as verified. A record at or below a path the walk left out,
|
||||||
// queued for the update phase, which writes the new disk_size. A record
|
// because -x stopped at it, it is a default exclude, or it is not on a
|
||||||
// at or below a path the walk left out, because -x stopped at it, it is
|
// persistent filesystem, counts as outside the roots: it is not
|
||||||
// a default exclude, or it is not on a persistent filesystem, counts as
|
// deleted. That does not hold below a root at or inside that path,
|
||||||
// outside the roots: it is not deleted. That does not hold below a root
|
// since that root's own walk verifies the records below it; nor is a
|
||||||
// at or inside that path, since that root's own walk verifies the
|
// line printed for such a root.
|
||||||
// records below it; nor is a line printed for such a root.
|
|
||||||
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||||
// The records the last scan left under the roots estimate how many
|
// The records the last scan left under the roots estimate how many
|
||||||
// files the walk will find. A first scan has none, and no estimate.
|
// files the walk will find. A first scan has none, and no estimate.
|
||||||
@@ -554,13 +546,6 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
|||||||
delete(s.existing, ev.rec.path)
|
delete(s.existing, ev.rec.path)
|
||||||
|
|
||||||
s.st.unchanged++
|
s.st.unchanged++
|
||||||
|
|
||||||
// The space a file takes on disk can change while its size and
|
|
||||||
// mtime stay the same. ZFS, for one, counts a new file's blocks
|
|
||||||
// up to seconds after the write.
|
|
||||||
if old.diskSize != ev.rec.diskSize {
|
|
||||||
s.diskSizes = append(s.diskSizes, ev.rec)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
prog.finish()
|
prog.finish()
|
||||||
@@ -710,7 +695,6 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
|
|||||||
|
|
||||||
s.batch = append(s.batch, scanRec{
|
s.batch = append(s.batch, scanRec{
|
||||||
size: rec.size,
|
size: rec.size,
|
||||||
diskSize: rec.diskSize,
|
|
||||||
mtime: rec.mtime,
|
mtime: rec.mtime,
|
||||||
head: r.head,
|
head: r.head,
|
||||||
tail: r.tail,
|
tail: r.tail,
|
||||||
@@ -731,7 +715,7 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
err := applyBatch(ctx, s.db, s.batch, nil, nil, nil)
|
err := applyBatch(ctx, s.db, s.batch, nil, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -742,10 +726,9 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// updatePhase writes the scan's tail under one progress display: the
|
// updatePhase writes the scan's tail under one progress display: the
|
||||||
// final partial batch of hashed records, the new disk_size of every
|
// final partial batch of hashed records, and deletions for every record
|
||||||
// unchanged file whose disk_size changed, and deletions for every
|
// the scan did not verify (vanished files, plus paths that failed to
|
||||||
// record the scan did not verify (vanished files, plus paths that
|
// stat or hash).
|
||||||
// failed to stat or hash).
|
|
||||||
func (s *scanState) updatePhase(ctx context.Context) error {
|
func (s *scanState) updatePhase(ctx context.Context) error {
|
||||||
deletes := make([]string, 0, len(s.existing))
|
deletes := make([]string, 0, len(s.existing))
|
||||||
for path := range s.existing {
|
for path := range s.existing {
|
||||||
@@ -757,7 +740,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
|||||||
|
|
||||||
s.st.removed = len(deletes)
|
s.st.removed = len(deletes)
|
||||||
|
|
||||||
total := len(s.batch) + len(s.diskSizes) + len(deletes)
|
total := len(s.batch) + len(deletes)
|
||||||
prog := newProgress(progressParams{
|
prog := newProgress(progressParams{
|
||||||
label: scanPhase("update"), unit: scanUnit, total: int64(total),
|
label: scanPhase("update"), unit: scanUnit, total: int64(total),
|
||||||
stalledOn: "db",
|
stalledOn: "db",
|
||||||
@@ -772,11 +755,6 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
|||||||
|
|
||||||
s.batch = nil
|
s.batch = nil
|
||||||
|
|
||||||
err = applyDiskSizes(ctx, s.db, s.diskSizes, prog)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
return applyChanges(ctx, s.db, nil, deletes, prog)
|
return applyChanges(ctx, s.db, nil, deletes, prog)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -896,18 +874,9 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
|||||||
|
|
||||||
// operandWarning returns the one-line warning for an operand the walk
|
// operandWarning returns the one-line warning for an operand the walk
|
||||||
// does not start from, naming the path and what it is, or "" for one it
|
// does not start from, naming the path and what it is, or "" for one it
|
||||||
// does: a regular file, or a directory not named .zfs, that is not
|
// does: a regular file, or a directory not named .zfs. Symlinks are
|
||||||
// inside a directory named .zfs. resolveRoots has already followed the
|
// never followed, including as operands.
|
||||||
// operand's symlinks, so a symlink here is an operand replaced by one
|
|
||||||
// after it was resolved, and is not followed.
|
|
||||||
func operandWarning(root string, fi fs.FileInfo) string {
|
func operandWarning(root string, fi fs.FileInfo) string {
|
||||||
// The walk never enters a .zfs directory, so it starts from nothing
|
|
||||||
// inside one either.
|
|
||||||
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
|
|
||||||
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
|
|
||||||
root)
|
|
||||||
}
|
|
||||||
|
|
||||||
var kind string
|
var kind string
|
||||||
|
|
||||||
switch mode := fi.Mode(); {
|
switch mode := fi.Mode(); {
|
||||||
@@ -978,7 +947,6 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
|
|||||||
rec := fileRec{
|
rec := fileRec{
|
||||||
path: root,
|
path: root,
|
||||||
size: fi.Size(),
|
size: fi.Size(),
|
||||||
diskSize: diskSizeOfInfo(fi),
|
|
||||||
mtime: fi.ModTime(),
|
mtime: fi.ModTime(),
|
||||||
dev: dev,
|
dev: dev,
|
||||||
ino: ino,
|
ino: ino,
|
||||||
@@ -1163,7 +1131,6 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
rec := fileRec{
|
rec := fileRec{
|
||||||
path: p,
|
path: p,
|
||||||
size: info.Size(),
|
size: info.Size(),
|
||||||
diskSize: diskSizeOfInfo(info),
|
|
||||||
mtime: info.ModTime(),
|
mtime: info.ModTime(),
|
||||||
dev: dev,
|
dev: dev,
|
||||||
ino: ino,
|
ino: ino,
|
||||||
@@ -1258,23 +1225,6 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
|
|||||||
return statDev(st), st.Ino
|
return statDev(st), st.Ino
|
||||||
}
|
}
|
||||||
|
|
||||||
// statBlockSize is the unit of st_blocks in an lstat result on Linux and
|
|
||||||
// macOS, whatever the filesystem's own block size.
|
|
||||||
const statBlockSize = 512
|
|
||||||
|
|
||||||
// diskSizeOfInfo returns the bytes a file takes on disk: its allocated
|
|
||||||
// blocks, st_blocks × 512. On a compressed or sparse file it is less
|
|
||||||
// than the file's size. It is 0 when the platform exposes no block
|
|
||||||
// count.
|
|
||||||
func diskSizeOfInfo(fi fs.FileInfo) int64 {
|
|
||||||
st, ok := fi.Sys().(*syscall.Stat_t)
|
|
||||||
if !ok {
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
return st.Blocks * statBlockSize
|
|
||||||
}
|
|
||||||
|
|
||||||
// hashResult carries the hashes computed for one inode run (or the
|
// hashResult carries the hashes computed for one inode run (or the
|
||||||
// error that prevented computing them) from the pool's workers to the
|
// error that prevented computing them) from the pool's workers to the
|
||||||
// hash phase. Its run leaves out the paths found removed.
|
// hash phase. Its run leaves out the paths found removed.
|
||||||
|
|||||||
+34
-219
@@ -42,30 +42,6 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
|
|||||||
return p
|
return p
|
||||||
}
|
}
|
||||||
|
|
||||||
// diskSizeOf returns the bytes the file at path takes on disk, from its
|
|
||||||
// own lstat: st_blocks counts 512-byte units. ZFS counts a write's
|
|
||||||
// blocks only when it commits the write, up to seconds later, and one
|
|
||||||
// commit can take the file through several growing sizes. So this value
|
|
||||||
// can change at any moment. Neither fsync nor unix.Sync hastens that,
|
|
||||||
// but a unix.Sync call often takes long enough that the blocks are
|
|
||||||
// counted when it returns. A test that needs the value to change waits
|
|
||||||
// for the change. A test that compares it with a scan's disk_size reads
|
|
||||||
// it before and after the scan. The rewrite test rescans until the two
|
|
||||||
// reads match, then compares exactly. The other two tests fail only on
|
|
||||||
// a disk_size outside the two reads.
|
|
||||||
func diskSizeOf(t *testing.T, path string) int64 {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
var st syscall.Stat_t
|
|
||||||
|
|
||||||
err := syscall.Lstat(path, &st)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return st.Blocks * 512
|
|
||||||
}
|
|
||||||
|
|
||||||
// pattern returns n bytes of deterministic content seeded by tag.
|
// pattern returns n bytes of deterministic content seeded by tag.
|
||||||
func pattern(tag byte, n int) []byte {
|
func pattern(tag byte, n int) []byte {
|
||||||
data := make([]byte, n)
|
data := make([]byte, n)
|
||||||
@@ -1088,13 +1064,12 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// buildSmokeTree recreates the tree of the test under README
|
// buildSmokeTree recreates the README smoke-test filesystem layout
|
||||||
// "Definition of done" with deterministic content and returns the tree
|
// with deterministic content and returns the tree root.
|
||||||
// root.
|
|
||||||
func buildSmokeTree(t *testing.T) string {
|
func buildSmokeTree(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
dir := realTempDir(t)
|
dir := t.TempDir()
|
||||||
one := pattern(10, 2000)
|
one := pattern(10, 2000)
|
||||||
f1 := pattern(30, 3000)
|
f1 := pattern(30, 3000)
|
||||||
f2 := pattern(40, 100)
|
f2 := pattern(40, 100)
|
||||||
@@ -1253,9 +1228,9 @@ func TestScanPipeline(t *testing.T) {
|
|||||||
// TestScanProgressNamesEachPhase checks the plain progress lines of a
|
// TestScanProgressNamesEachPhase checks the plain progress lines of a
|
||||||
// first scan, and of a rescan of one operand holding two of the
|
// first scan, and of a rescan of one operand holding two of the
|
||||||
// database's records. Each phase's first line names it with its place
|
// database's records. Each phase's first line names it with its place
|
||||||
// among the four: load gets the number of records before reading them,
|
// among the four: load counts the records before reading them, and the
|
||||||
// and the walk estimates its total from the records under its operand,
|
// walk estimates its total from the records under its operand, or says
|
||||||
// or says its time remaining is unknown when there are none.
|
// its time remaining is unknown when there are none.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
func TestScanProgressNamesEachPhase(t *testing.T) {
|
func TestScanProgressNamesEachPhase(t *testing.T) {
|
||||||
@@ -1441,159 +1416,6 @@ func TestScanStoresRawHashes(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestScanStoresDiskSize scans a sparse file, which takes less space on
|
|
||||||
// disk than its length, once through its directory and once as the
|
|
||||||
// operand itself. Each scan stores the disk_size of the file's lstat.
|
|
||||||
func TestScanStoresDiskSize(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
const length = 1024 * 1024
|
|
||||||
|
|
||||||
p := sparseFile(t, t.TempDir(), "sparse", length)
|
|
||||||
pokeAt(t, p, 0, []byte("x"))
|
|
||||||
|
|
||||||
unix.Sync() // see diskSizeOf
|
|
||||||
|
|
||||||
size := diskSizeOf(t, p)
|
|
||||||
if size == 0 || size >= length {
|
|
||||||
t.Fatalf("the file takes %d bytes on disk, want more than none "+
|
|
||||||
"and fewer than its length %d", size, length)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, operand := range []string{filepath.Dir(p), p} {
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
// ZFS may count the blocks during the scan; see diskSizeOf.
|
|
||||||
before := diskSizeOf(t, p)
|
|
||||||
|
|
||||||
syncTree(t, db, operand)
|
|
||||||
|
|
||||||
after := diskSizeOf(t, p)
|
|
||||||
|
|
||||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
|
||||||
if got < before || got > after {
|
|
||||||
t.Errorf("scan of %s: disk_size = %d, want %d to %d",
|
|
||||||
operand, got, before, after)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSyncScanRewriteUpdatesDiskSize writes into the hole of a scanned
|
|
||||||
// sparse file and moves its mtime on. The rescan hashes the file again
|
|
||||||
// and rewrites its record with the new disk_size.
|
|
||||||
func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
db := openTestDB(t)
|
|
||||||
p := sparseFile(t, dir, "sparse", 1024*1024)
|
|
||||||
pokeAt(t, p, 0, []byte("x"))
|
|
||||||
|
|
||||||
syncTree(t, db, dir)
|
|
||||||
|
|
||||||
recorded := recordByPath(t, dbRecords(t, db), p).diskSize
|
|
||||||
|
|
||||||
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
|
|
||||||
|
|
||||||
// The write may leave the mtime the scan recorded; a later one makes
|
|
||||||
// the rescan read the file again.
|
|
||||||
later := time.Now().Add(time.Hour)
|
|
||||||
|
|
||||||
err := os.Chtimes(p, later, later)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Wait for ZFS to count the write; see diskSizeOf.
|
|
||||||
deadline := time.Now().Add(30 * time.Second)
|
|
||||||
for diskSizeOf(t, p) == recorded {
|
|
||||||
if time.Now().After(deadline) {
|
|
||||||
t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
|
|
||||||
"after the write into its hole", recorded)
|
|
||||||
}
|
|
||||||
|
|
||||||
unix.Sync()
|
|
||||||
}
|
|
||||||
|
|
||||||
// ZFS may still be counting, so the size may change during a rescan.
|
|
||||||
// Only a rescan with the same size on both sides of it can be compared
|
|
||||||
// exactly. A rescan that keeps the old size still fails, because the
|
|
||||||
// wait made before differ from it.
|
|
||||||
for {
|
|
||||||
before := diskSizeOf(t, p)
|
|
||||||
|
|
||||||
st := syncTree(t, db, dir)
|
|
||||||
|
|
||||||
after := diskSizeOf(t, p)
|
|
||||||
|
|
||||||
if st != (scanStats{walked: 1, updated: 1}) {
|
|
||||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
|
||||||
}
|
|
||||||
|
|
||||||
if before == after {
|
|
||||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
|
||||||
if got != after {
|
|
||||||
t.Errorf("disk_size = %d, want %d", got, after)
|
|
||||||
}
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
if time.Now().After(deadline) {
|
|
||||||
t.Fatalf("the size on disk was still changing 30 seconds after "+
|
|
||||||
"the write into its hole: %d, then %d", before, after)
|
|
||||||
}
|
|
||||||
|
|
||||||
// A later mtime makes the next rescan read the file again.
|
|
||||||
later = later.Add(time.Second)
|
|
||||||
|
|
||||||
err = os.Chtimes(p, later, later)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSyncScanUnchangedUpdatesDiskSize gives a file's record a
|
|
||||||
// disk_size the file does not have, as when ZFS had not yet counted the
|
|
||||||
// file's blocks at the last scan. The rescan writes the file's own
|
|
||||||
// disk_size into the record without reading the file again.
|
|
||||||
func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
db := openTestDB(t)
|
|
||||||
p := writeFile(t, dir, "a.bin", pattern(1, 500))
|
|
||||||
|
|
||||||
syncTree(t, db, dir)
|
|
||||||
|
|
||||||
// lstat counts 512-byte units, so no file takes 1 byte on disk.
|
|
||||||
_, err := db.ExecContext(t.Context(),
|
|
||||||
"UPDATE files SET disk_size = 1 WHERE path = ?", []byte(p))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
unix.Sync() // see diskSizeOf
|
|
||||||
|
|
||||||
// ZFS may count the blocks during the rescan; see diskSizeOf.
|
|
||||||
before := diskSizeOf(t, p)
|
|
||||||
|
|
||||||
// A file read again would count as updated.
|
|
||||||
st := syncTree(t, db, dir)
|
|
||||||
|
|
||||||
after := diskSizeOf(t, p)
|
|
||||||
|
|
||||||
if st != (scanStats{walked: 1, unchanged: 1}) {
|
|
||||||
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
|
||||||
if got < before || got > after {
|
|
||||||
t.Errorf("disk_size = %d, want %d to %d", got, before, after)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
|
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
|
||||||
// size with an mtime later in the same second as the recorded one: the
|
// size with an mtime later in the same second as the recorded one: the
|
||||||
// next scan must notice the change and re-hash the file. The operand is
|
// next scan must notice the change and re-hash the file. The operand is
|
||||||
@@ -2310,24 +2132,33 @@ func TestUnderRoot(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
||||||
// database, and in one case also a hard link to the database file. The
|
// database, reached directly and through a symlinked directory. The
|
||||||
// walk knows the database's files by device ID and inode number, so it
|
// walk passes over the database file, its -wal and -shm files and its
|
||||||
// passes over the database file, its -wal and -shm files, its .lock
|
// .lock file with no line and no count, and the record an earlier scan
|
||||||
// file and the hard link with no line and no count. The record an
|
// left for the .lock file is deleted. A rescan of the unchanged tree
|
||||||
// earlier scan left for the .lock file is deleted. A rescan of the
|
// updates nothing, and report lists none of the database's files.
|
||||||
// unchanged tree updates nothing, and report lists none of the
|
|
||||||
// database's files.
|
|
||||||
func TestRunScanPassesOverDatabase(t *testing.T) {
|
func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||||
for _, viaHardLink := range []bool{false, true} {
|
for _, viaSymlink := range []bool{false, true} {
|
||||||
t.Run(fmt.Sprintf("via hard link %v", viaHardLink), func(t *testing.T) {
|
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
|
||||||
tree := realTempDir(t)
|
base := t.TempDir()
|
||||||
hardLink := filepath.Join(tree, "hardlink")
|
tree := filepath.Join(base, "real", "tree")
|
||||||
|
operand := tree
|
||||||
|
|
||||||
writeFile(t, tree, "a", pattern(1, 100))
|
writeFile(t, tree, "a", pattern(1, 100))
|
||||||
writeFile(t, tree, "b", pattern(1, 100))
|
writeFile(t, tree, "b", pattern(1, 100))
|
||||||
writeFile(t, tree, "empty1", nil)
|
writeFile(t, tree, "empty1", nil)
|
||||||
writeFile(t, tree, "empty2", nil)
|
writeFile(t, tree, "empty2", nil)
|
||||||
|
|
||||||
|
if viaSymlink {
|
||||||
|
err := os.Symlink(filepath.Join(base, "real"),
|
||||||
|
filepath.Join(base, "link"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
operand = filepath.Join(base, "link", "tree")
|
||||||
|
}
|
||||||
|
|
||||||
dbPath := filepath.Join(tree, "db.sqlite")
|
dbPath := filepath.Join(tree, "db.sqlite")
|
||||||
t.Setenv(databaseEnv, dbPath)
|
t.Setenv(databaseEnv, dbPath)
|
||||||
|
|
||||||
@@ -2337,7 +2168,7 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
err = applyChanges(t.Context(), db, []scanRec{
|
err = applyChanges(t.Context(), db, []scanRec{
|
||||||
{path: filepath.Join(tree, "db.sqlite.lock")},
|
{path: filepath.Join(operand, "db.sqlite.lock")},
|
||||||
}, nil, nil)
|
}, nil, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -2345,29 +2176,16 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
|||||||
|
|
||||||
closeScanDatabase(t.Context(), db, dbPath)
|
closeScanDatabase(t.Context(), db, dbPath)
|
||||||
|
|
||||||
if viaHardLink {
|
assertScanSummary(t,
|
||||||
err = os.Link(dbPath, hardLink)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stderr := assertScanSummary(t,
|
|
||||||
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
||||||
tree)
|
operand)
|
||||||
stderr += assertScanSummary(t,
|
assertScanSummary(t,
|
||||||
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
||||||
tree)
|
operand)
|
||||||
|
|
||||||
if strings.Contains(stderr, hardLink) {
|
|
||||||
t.Errorf("scan stderr = %q, want no line naming %s",
|
|
||||||
stderr, hardLink)
|
|
||||||
}
|
|
||||||
|
|
||||||
report := runStdout(t, cmdReport)
|
report := runStdout(t, cmdReport)
|
||||||
if strings.Contains(report, "db.sqlite") ||
|
if strings.Contains(report, "db.sqlite") ||
|
||||||
strings.Contains(report, hardLink) ||
|
!strings.Contains(report, filepath.Join(operand, "empty2")) {
|
||||||
!strings.Contains(report, filepath.Join(tree, "empty2")) {
|
|
||||||
t.Errorf("report = %q, want the empty files and none of "+
|
t.Errorf("report = %q, want the empty files and none of "+
|
||||||
"the database's files", report)
|
"the database's files", report)
|
||||||
}
|
}
|
||||||
@@ -2392,9 +2210,8 @@ func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
|
|||||||
|
|
||||||
// assertScanSummary scans operands and fails the test unless the
|
// assertScanSummary scans operands and fails the test unless the
|
||||||
// summary line gives counts with nothing skipped, and the line naming
|
// summary line gives counts with nothing skipped, and the line naming
|
||||||
// the database is the only one that names one of its files. It returns
|
// the database is the only one that names one of its files.
|
||||||
// everything the scan printed to stderr.
|
func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
||||||
func assertScanSummary(t *testing.T, counts string, operands ...string) string {
|
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
stderr := scanOK(t, operands...)
|
stderr := scanOK(t, operands...)
|
||||||
@@ -2408,6 +2225,4 @@ func assertScanSummary(t *testing.T, counts string, operands ...string) string {
|
|||||||
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
||||||
"database to name one of its files", operands, stderr)
|
"database to name one of its files", operands, stderr)
|
||||||
}
|
}
|
||||||
|
|
||||||
return stderr
|
|
||||||
}
|
}
|
||||||
@@ -22,10 +22,6 @@ type treeNode struct {
|
|||||||
digest [sha256.Size]byte
|
digest [sha256.Size]byte
|
||||||
fileCount int64
|
fileCount int64
|
||||||
totalSize int64
|
totalSize int64
|
||||||
// diskSize is the bytes the directory's files take on disk. Two
|
|
||||||
// copies of one tree have the same totalSize, but their diskSize
|
|
||||||
// can differ.
|
|
||||||
diskSize int64
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// treesPhase names one phase of trees with its place among all of them,
|
// treesPhase names one phase of trees with its place among all of them,
|
||||||
@@ -44,11 +40,11 @@ func treesPhase(name string) string {
|
|||||||
// database problem, including a missing database, is fatal.
|
// database problem, including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in four phases, each under a progress display of its
|
// The work runs in four phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database), read (read the number of
|
// own on stderr: open (open the database), read (count the records,
|
||||||
// records from the totals table, then read the records into the
|
// then read them into the hierarchy), group (group the directories by
|
||||||
// hierarchy), group (group the directories by digest) and write (write
|
// digest) and write (write the groups to stdout). Each display ends
|
||||||
// the groups to stdout). Each display ends before the next phase
|
// before the next phase starts, and before an error is returned to be
|
||||||
// starts, and before an error is returned to be printed.
|
// printed.
|
||||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -143,16 +139,15 @@ func readTree(ctx context.Context, db *sql.DB,
|
|||||||
|
|
||||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||||
// a row for every member of each group but its first, counting each
|
// a row for every member of each group but its first, counting each
|
||||||
// group on prog once its rows are written. The disk_size column is the
|
// group on prog once its rows are written. stdout may be the terminal
|
||||||
// dupe's own. stdout may be the terminal prog is drawn on, so the lines
|
// prog is drawn on, so the lines go through prog's writer. It returns
|
||||||
// go through prog's writer. It returns the number of rows (dupe trees)
|
// the number of rows (dupe trees) and the bytes they could reclaim.
|
||||||
// and the bytes they could reclaim.
|
|
||||||
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||||
prog *progress,
|
prog *progress,
|
||||||
) (int, int64, error) {
|
) (int, int64, error) {
|
||||||
out := prog.writer(stdout)
|
out := prog.writer(stdout)
|
||||||
|
|
||||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
|
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||||
}
|
}
|
||||||
@@ -164,9 +159,9 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
|||||||
for _, g := range dupes {
|
for _, g := range dupes {
|
||||||
first := g[0]
|
first := g[0]
|
||||||
for _, n := range g[1:] {
|
for _, n := range g[1:] {
|
||||||
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\t%d\n",
|
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n",
|
||||||
escapePath(first.path), escapePath(n.path),
|
escapePath(first.path), escapePath(n.path),
|
||||||
first.fileCount, first.totalSize, n.diskSize)
|
first.fileCount, first.totalSize)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||||
}
|
}
|
||||||
@@ -237,7 +232,6 @@ func (b *treeBuilder) add(r scanRec) {
|
|||||||
dir.entries = append(dir.entries, fileEntry(name, r))
|
dir.entries = append(dir.entries, fileEntry(name, r))
|
||||||
dir.fileCount++
|
dir.fileCount++
|
||||||
dir.totalSize += r.size
|
dir.totalSize += r.size
|
||||||
dir.diskSize += r.diskSize
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// openDir starts the directory called name inside the innermost open
|
// openDir starts the directory called name inside the innermost open
|
||||||
@@ -276,7 +270,6 @@ func (b *treeBuilder) closeTo(n int) {
|
|||||||
"d\x00"+name+"\x00"+string(dir.digest[:]))
|
"d\x00"+name+"\x00"+string(dir.digest[:]))
|
||||||
dir.parent.fileCount += dir.fileCount
|
dir.parent.fileCount += dir.fileCount
|
||||||
dir.parent.totalSize += dir.totalSize
|
dir.parent.totalSize += dir.totalSize
|
||||||
dir.parent.diskSize += dir.diskSize
|
|
||||||
b.dirs = append(b.dirs, dir)
|
b.dirs = append(b.dirs, dir)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
+13
-42
@@ -15,26 +15,20 @@ import (
|
|||||||
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
|
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
|
||||||
"remain unknown"
|
"remain unknown"
|
||||||
|
|
||||||
// smokeTreeRecs mirrors the tree of the test under README "Definition
|
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
|
||||||
// of done": /d/t1 and /d/t2 are identical, /d/t3 differs from them only
|
// /d/t2 are identical, /d/t3 differs from them only by one filename.
|
||||||
// by one filename. Each file takes one 4 KiB block on disk.
|
|
||||||
func smokeTreeRecs() []scanRec {
|
func smokeTreeRecs() []scanRec {
|
||||||
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
|
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
|
||||||
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
|
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
|
||||||
|
|
||||||
return []scanRec{
|
return []scanRec{
|
||||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"},
|
||||||
content: f1Content, path: "/d/t1/f1"},
|
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"},
|
||||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t2/f1"},
|
||||||
content: f2Content, path: "/d/t1/sub/f2"},
|
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t2/sub/f2"},
|
||||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t3/f1"},
|
||||||
content: f1Content, path: "/d/t2/f1"},
|
{size: 100, head: f2Head, tail: f2Tail, content: f2Content,
|
||||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
path: "/d/t3/sub/f2renamed"},
|
||||||
content: f2Content, path: "/d/t2/sub/f2"},
|
|
||||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
|
||||||
content: f1Content, path: "/d/t3/f1"},
|
|
||||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
|
||||||
content: f2Content, path: "/d/t3/sub/f2renamed"},
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -204,33 +198,13 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
|||||||
code, exitOK, stderr.String())
|
code, exitOK, stderr.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
want := "first\tdupe\tfiles\tsize\n" +
|
||||||
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n"
|
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\n"
|
||||||
if got := stdout.String(); got != want {
|
if got := stdout.String(); got != want {
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
|
||||||
// take different space on disk. The row's disk_size is the sum over the
|
|
||||||
// dupe copy's files, those in its subdirectory included.
|
|
||||||
func TestRunTreesDiskSize(t *testing.T) {
|
|
||||||
f, g := testHash("f"), testHash("g")
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 10, diskSize: 4096, head: f, tail: f, content: f, path: "/a/f"},
|
|
||||||
{size: 20, diskSize: 4096, head: g, tail: g, content: g, path: "/a/sub/g"},
|
|
||||||
{size: 10, diskSize: 512, head: f, tail: f, content: f, path: "/b/f"},
|
|
||||||
{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
|
||||||
|
|
||||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n"
|
|
||||||
if got := runStdout(t, cmdTrees); got != want {
|
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
|
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
|
||||||
// records with stderr not a terminal, and checks every line it prints
|
// records with stderr not a terminal, and checks every line it prints
|
||||||
// there. After the line naming the database, each phase prints a line
|
// there. After the line naming the database, each phase prints a line
|
||||||
@@ -252,8 +226,7 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
|||||||
code, exitOK, stderr())
|
code, exitOK, stderr())
|
||||||
}
|
}
|
||||||
|
|
||||||
report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
|
||||||
"/d/t1\t/d/t2\t2\t3100\t8192\n"
|
|
||||||
if got := stdout.String(); got != report {
|
if got := stdout.String(); got != report {
|
||||||
t.Errorf("stdout = %q, want %q", got, report)
|
t.Errorf("stdout = %q, want %q", got, report)
|
||||||
}
|
}
|
||||||
@@ -318,9 +291,7 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
|||||||
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
||||||
}
|
}
|
||||||
|
|
||||||
want := []string{
|
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
|
||||||
"first\tdupe\tfiles\tsize\tdisk_size", "/d/t1\t/d/t2\t2\t3100\t8192",
|
|
||||||
}
|
|
||||||
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
|
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
|
||||||
t.Errorf("terminal shows lines %q, want %q first", shown, want)
|
t.Errorf("terminal shows lines %q, want %q first", shown, want)
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in new issue
Block a user