Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
82774d41e9 |
@@ -20,7 +20,7 @@
|
|||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
- Take the next issue from the `1.0.0` milestone,
|
- Take the next issue from the `1.0.0` milestone:
|
||||||
https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated
|
https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated
|
||||||
here, because a copy drifts as soon as the tracker moves.
|
here, because a copy drifts as soon as the tracker moves.
|
||||||
|
|
||||||
@@ -29,32 +29,33 @@
|
|||||||
Newest first. An entry ends with its date and its issue or branch, where those
|
Newest first. An entry ends with its date and its issue or branch, where those
|
||||||
are known.
|
are known.
|
||||||
|
|
||||||
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
|
- `TODO.md` rewritten from scratch by the same rules as `README.md`: short
|
||||||
sentence. Every completed step is kept, with its date and its issue or branch.
|
sentences, lists as bullets, nothing a newcomer reads twice. Every completed
|
||||||
The policy-compliance checklist of 2026-07-22 is folded into its step
|
step is kept, with its date and its issue or branch. The policy-compliance
|
||||||
(2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
checklist of 2026-07-22 is folded into its step (2026-10-26,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||||
|
|
||||||
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
|
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
|
||||||
it. It leads with what the program is and how to use it. A reference section
|
it. It leads with what the program is and how to use it. A reference section
|
||||||
covers every setting and every subcommand. Development and testing come last.
|
covers the settings, the database, the output formats and the exit codes;
|
||||||
Every statement was checked against the code on `next`. Three comments in
|
development and testing come last. Every statement was checked against the
|
||||||
`main_test.go` cite "Settings", where their rules now live (2026-10-26,
|
code on `next`. Three comments in `main_test.go` cite "Settings", where their
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/135)
|
rules now live (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||||
|
|
||||||
- A file or directory removed while the scan runs is not an error, in the walk
|
- A file or directory removed while the scan runs is not an error, in the walk
|
||||||
or in the hash phase. The scan prints no line for it and does not count it as
|
or in the hash phase. The scan prints no line for it and does not count it as
|
||||||
skipped. When the first of several hard links to a file is gone, the hash
|
skipped. When the first of several hard links to a file is gone, the hash
|
||||||
phase reads the next one instead. On a terminal every progress display starts
|
phase reads the next one instead. On a terminal every progress display starts
|
||||||
with the spinner, the hash phase's included. A phase with an exact total ends
|
with the spinner, the hash phase's included. A phase with an exact total ends
|
||||||
its line with its percent and a 10-column bar at the terminal's right edge.
|
its line with its percent and a 10-column bar at the terminal's right edge,
|
||||||
The bar no longer moves as the line's length changes. A line too wide for the
|
which no longer moves as the line's length changes. A line too wide for the
|
||||||
terminal now loses its rate before its bar. The time remaining is labelled
|
terminal now loses its rate before its bar. The time remaining is labelled
|
||||||
`remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133)
|
`remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133)
|
||||||
|
|
||||||
- The `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
|
- The `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
|
||||||
bytes of a SHA-256. That is half the size of the hex text they held. The
|
bytes of a SHA-256, half the size of the hex text they held. The hashes stay
|
||||||
hashes stay raw from the hash phase through the database to `report` and
|
raw from the hash phase through the database to `report` and `trees`. Reading
|
||||||
`trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26,
|
a record whose hash is not 32 bytes is an error (2026-10-26,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/134)
|
https://git.eeqj.de/sneak/sfdupes/issues/134)
|
||||||
|
|
||||||
- Every file the walk records gets three hashes in the hash phase. `head` and
|
- Every file the walk records gets three hashes in the hash phase. `head` and
|
||||||
@@ -62,9 +63,9 @@ are known.
|
|||||||
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
|
of the whole file under 100 MiB, and of 1 MiB at each gigabyte from 100 MiB
|
||||||
up. Gone are:
|
up. Gone are:
|
||||||
- the 10 MiB and 50 MiB thresholds;
|
- the 10 MiB and 50 MiB thresholds;
|
||||||
- the size census, which counted the sizes of every walked file and of every
|
- the size census: the sizes of every walked file and of every record
|
||||||
record outside the operands. A file was hashed only when the census showed
|
outside the operands, by which a file was hashed only when another file
|
||||||
another file of its size;
|
shared its size;
|
||||||
- the records without hashes;
|
- the records without hashes;
|
||||||
- the content phase.
|
- the content phase.
|
||||||
|
|
||||||
@@ -73,108 +74,72 @@ are known.
|
|||||||
with a content hash, since every record has all three (2026-10-25,
|
with a content hash, since every record has all three (2026-10-25,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/132)
|
https://git.eeqj.de/sneak/sfdupes/issues/132)
|
||||||
|
|
||||||
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` copied again
|
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` re-vendored from
|
||||||
from `sneak/prompts` `next` at `0653579`. Two steps now come before
|
`sneak/prompts` `next` at `0653579`. Two steps now come before
|
||||||
`ARG CACHEBUST`:
|
`ARG CACHEBUST`: the module download in the `lint`, `test` and `builder`
|
||||||
- the module download in the `lint`, `test` and `builder` stages of the
|
stages of the `Dockerfile`, and the package install in `builder`. Docker keeps
|
||||||
`Dockerfile`;
|
them until the base image, the install line, `go.mod` or `go.sum` changes.
|
||||||
- the package install in `builder`.
|
Every step that lints, tests or compiles still comes after it.
|
||||||
|
`golangci-lint run` also keeps the linter's own cache. README says the same
|
||||||
|
(2026-10-18, https://git.eeqj.de/sneak/sfdupes/issues/109)
|
||||||
|
|
||||||
Docker keeps them until one of these changes:
|
- Release rule set: until 1.0, each tip of `main` that passes the build gets the
|
||||||
- the base image;
|
next `v0.0.N` tag, so `go install sneak.berlin/go/sfdupes@latest` installs it.
|
||||||
- the install line;
|
`go.mod` retracts `v0.0.1`, the early build without a database. When the
|
||||||
- `go.mod` or `go.sum`.
|
|
||||||
|
|
||||||
Every step that checks or compiles the code still comes after
|
|
||||||
`ARG CACHEBUST`. `golangci-lint run` also keeps the linter's own cache.
|
|
||||||
README says the same (2026-10-18,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/109)
|
|
||||||
|
|
||||||
- Release rule set. Until 1.0, each tip of `main` that passes the build gets the
|
|
||||||
next `v0.0.N` tag. `go install sneak.berlin/go/sfdupes@latest` then installs
|
|
||||||
it. `go.mod` retracts `v0.0.1`, the early build without a database. When the
|
|
||||||
linker set no version, `sfdupes --version` prints the version Go recorded in
|
linker set no version, `sfdupes --version` prints the version Go recorded in
|
||||||
the binary. That is the tag, or the pseudo-version of a `go install` build. It
|
the binary: the tag, or the pseudo-version of a `go install` build. It prints
|
||||||
prints `dev` only when neither is known. README "Install" documented:
|
`dev` only when neither is known. README "Install" documented the rule, what
|
||||||
- the rule;
|
`@main` and `v0.0.1` give, and how to check an installed binary (2026-10-17,
|
||||||
- what `@main` and `v0.0.1` give;
|
https://git.eeqj.de/sneak/sfdupes/issues/116)
|
||||||
- how to check an installed binary.
|
|
||||||
|
|
||||||
(2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/116)
|
|
||||||
|
|
||||||
- `scan` creates the database's directory where the database path leads. The
|
- `scan` creates the database's directory where the database path leads. The
|
||||||
directory is the path with its last element cut off as text, so a `..` in it
|
directory is the path with its last element cut off as text, never cleaned. A
|
||||||
stays. A `..` after a symlinked directory therefore goes through the symlink,
|
`..` after a symlinked directory therefore goes through the symlink, as it
|
||||||
as it does when the file is opened. With
|
does when the file is opened. With
|
||||||
`SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a symlink to
|
`SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a symlink to
|
||||||
`/data/x`, it now creates `/data/new` and the database in it. It used to
|
`/data/x`, it now creates `/data/new` and the database in it. It used to
|
||||||
create `/srv/new` and then fail (2026-10-17,
|
create `/srv/new` and then fail (2026-10-17,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/119)
|
https://git.eeqj.de/sneak/sfdupes/issues/119)
|
||||||
|
|
||||||
- `scan` never records its own database. Once the database and its `.lock` file
|
- `scan` never records its own database. Once the database and its `.lock` file
|
||||||
are open, it notes the device and inode of each of these:
|
are open, it notes the device and inode of the database file, its `-wal` and
|
||||||
- the database file;
|
`-shm` files and the `.lock` file. The walk passes over a file with one of
|
||||||
- its `-wal` and `-shm` files;
|
them wherever it reaches it, through a symlinked directory included, printing
|
||||||
- the `.lock` file.
|
no line and counting nothing. A record an earlier scan left for one of them is
|
||||||
|
deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
|
||||||
|
|
||||||
The walk passes over any file with the device and inode of one of them,
|
- `scan` never records or reads anything on a filesystem that is not persistent:
|
||||||
however it reaches the file. It prints no line for such a file and counts
|
`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types. It
|
||||||
nothing. A record an earlier scan left for one of them is deleted
|
matches them by device ID, however the walk or the content phase reaches a
|
||||||
(2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
|
path. As the scan starts, it lists the mounted filesystems of those types,
|
||||||
|
from `/proc/self/mountinfo` on Linux and from the mount list on macOS. No flag
|
||||||
- `scan` never records or reads anything on a filesystem that is not persistent,
|
turns that off. Below an operand, the walk does not enter the default excludes
|
||||||
such as `tmpfs` or `proc`. Such a filesystem is one the kernel makes up or
|
either, so `sfdupes scan /` skips the kernel's view of the running system and
|
||||||
keeps in memory. `scan` matches them by device ID, however the walk or the
|
the temporary directories. On Linux those are `/proc`, `/sys`, `/dev`, `/run`,
|
||||||
content phase reaches a path. As the scan starts, it lists the mounted
|
`/tmp` and `/var/tmp`. On macOS they are `/dev`, `/private/tmp`,
|
||||||
filesystems of those types, from `/proc/self/mountinfo` on Linux and from the
|
`/private/var/tmp`, `/private/var/vm`, `/System/Volumes/VM` and
|
||||||
mount list on macOS. No flag turns that off. Below an operand, the walk does
|
`/System/Volumes/Data`. A Windows list exists in the code but is not used. An
|
||||||
not enter the default excludes either. So `sfdupes scan /` skips the kernel's
|
operand at or inside a default exclude is still walked. The scan prints a line
|
||||||
view of the running system and the temporary directories. On Linux those are:
|
for each path it leaves out, without counting it as skipped, and the records
|
||||||
|
there count as outside the scanned operands, as for `-x`.
|
||||||
```
|
`--no-default-excludes` turns off the default excludes only, and `scan --help`
|
||||||
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
lists them (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/124)
|
||||||
```
|
|
||||||
|
|
||||||
On macOS they are:
|
|
||||||
|
|
||||||
```
|
|
||||||
/dev, /private/tmp, /private/var/tmp, /private/var/vm
|
|
||||||
/System/Volumes/VM, /System/Volumes/Data
|
|
||||||
```
|
|
||||||
|
|
||||||
A Windows list exists in the code but is not used. An operand at or inside a
|
|
||||||
default exclude is still walked. The scan prints a line for each path it
|
|
||||||
leaves out, without counting it as skipped. The records there count as
|
|
||||||
outside the scanned operands, as for `-x`. `--no-default-excludes` turns off
|
|
||||||
the default excludes only, and `scan --help` lists them (2026-10-17,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
|
||||||
|
|
||||||
- The rate a progress display shows covers the items completed over the last 5
|
- The rate a progress display shows covers the items completed over the last 5
|
||||||
seconds. When stderr is not a terminal, it covers the items since the line
|
seconds, or since the line before when stderr is not a terminal. It falls to 0
|
||||||
before. It falls to 0 while nothing completes. A burst after a wait no longer
|
while nothing completes, and a burst after a wait no longer reads as a steady
|
||||||
reads as a steady rate of over 1,000 files a second. The time remaining still
|
rate of over 1,000 files a second. The time remaining still comes from the
|
||||||
comes from the phase's average. Once no item has completed for 10 seconds, the
|
phase's average. Once no item has completed for 10 seconds, the word `stalled`
|
||||||
word `stalled` and where the phase waits take the rate's place, as
|
and where the phase waits take the rate's place, as `stalled disk/db`. The
|
||||||
`stalled disk/db`. The places are:
|
places are `disk`, `db`, `out` for stdout and `cpu`, or two of them for a
|
||||||
- `disk`;
|
phase that may wait on either. Each phase's word is set in the code that
|
||||||
- `db`;
|
defines the phase. A line too wide for the terminal loses its bar first, then
|
||||||
- `out`, for stdout;
|
its rate or those words, so the phase, progress, elapsed time and ETA stay
|
||||||
- `cpu`.
|
whole. On an 80-column terminal, at counts like those of README's example, the
|
||||||
|
|
||||||
A phase that may wait in two places names both. Each phase's word is set in
|
|
||||||
the code that defines the phase. A line too wide for the terminal loses its
|
|
||||||
bar first, then its rate or those words. Four parts stay whole:
|
|
||||||
- the phase;
|
|
||||||
- the progress;
|
|
||||||
- the elapsed time;
|
|
||||||
- the ETA, the time remaining.
|
|
||||||
|
|
||||||
On an 80-column terminal, at counts like those of README's example, the
|
|
||||||
content phase's displays for its checks and its reads lost those words. That
|
content phase's displays for its checks and its reads lost those words. That
|
||||||
happened for most elapsed times and ETAs in minutes. Every other phase kept
|
happened for most elapsed times and ETAs in minutes. Every other phase kept
|
||||||
them whenever the elapsed time and the ETA were both in minutes. The content
|
them whenever the elapsed time and the ETA were both in minutes. The content
|
||||||
phase writes its last partial batch, the records not yet written, under a
|
phase's last partial batch is written under a display of its own (2026-10-16,
|
||||||
display of its own (2026-10-16,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||||
|
|
||||||
- New subcommand `info` prints the database's details on stdout, one per line as
|
- New subcommand `info` prints the database's details on stdout, one per line as
|
||||||
@@ -183,15 +148,13 @@ are known.
|
|||||||
- the sizes of the database file and of its `-wal` file, when there is one;
|
- the sizes of the database file and of its `-wal` file, when there is one;
|
||||||
- when the database file was last changed;
|
- when the database file was last changed;
|
||||||
- the schema version;
|
- the schema version;
|
||||||
- the number of records;
|
- the number of records, how many have `head` and `tail` hashes, and how
|
||||||
- how many records have `head` and `tail` hashes;
|
many have a content hash;
|
||||||
- how many records have a content hash;
|
|
||||||
- the sizes of all records added up.
|
- the sizes of all records added up.
|
||||||
|
|
||||||
It opens the database as `report` does and shows two phases on stderr,
|
It opens the database as `report` does and shows two phases on stderr,
|
||||||
`[1/2] open` and `[2/2] count`. The schema version and the counts come from
|
`[1/2] open` and `[2/2] count`. The schema version and the counts come from
|
||||||
one read-only transaction, a read of the database at one moment. The counts
|
one read-only transaction, the counts from one query (2026-10-16,
|
||||||
come from one query (2026-10-16,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/117)
|
https://git.eeqj.de/sneak/sfdupes/issues/117)
|
||||||
|
|
||||||
- `report` shows its work on stderr in two phases, `[1/2] open` and
|
- `report` shows its work on stderr in two phases, `[1/2] open` and
|
||||||
@@ -200,75 +163,68 @@ are known.
|
|||||||
opening the database, also while it waits on a busy one. `read` counts the
|
opening the database, also while it waits on a busy one. `read` counts the
|
||||||
records, then reads them against that count in signature order. SQLite takes
|
records, then reads them against that count in signature order. SQLite takes
|
||||||
that order from the `files_signature` index without sorting, so the first
|
that order from the `files_signature` index without sorting, so the first
|
||||||
record comes at once. `report` forms the duplicate groups itself and holds
|
record comes at once. `report` forms the duplicate groups itself, holding only
|
||||||
only those of the size it is reading. It writes them ordered by first path
|
those of the size it is reading, and writes them ordered by first path before
|
||||||
before it reads the next size. stdout and the summary line are unchanged, and
|
it reads the next size. stdout and the summary line are unchanged, and SQLite
|
||||||
SQLite no longer needs a temporary file to sort in (2026-10-16,
|
no longer needs a temporary file to sort in (2026-10-16,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/114)
|
https://git.eeqj.de/sneak/sfdupes/issues/114)
|
||||||
|
|
||||||
- `trees` shows its work on stderr in four phases, from `[1/4] open` to
|
- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
|
||||||
`[4/4] write`, each with the progress display `scan` uses. The display runs
|
`[3/4] group` and `[4/4] write`, each with the progress display `scan` uses.
|
||||||
from the moment `trees` starts until the summary line. `open` says it is
|
The display runs from the moment `trees` starts until the summary line. `open`
|
||||||
opening the database, also while it waits on a busy one. `read` counts the
|
says it is opening the database, also while it waits on a busy one. `read`
|
||||||
records, then reads them against that count. `group` counts the directories it
|
counts the records, then reads them against that count. `group` counts the
|
||||||
groups, and `write` the groups it writes. A display's rate names what its
|
directories it groups, and `write` the groups it writes. A display's rate
|
||||||
phase counts, such as records or groups. stdout and the summary line are
|
names what its phase counts: files, records, directories or groups. stdout and
|
||||||
unchanged. When stdout and stderr are the same file, as one terminal is, no
|
the summary line are unchanged. When stdout and stderr are the same file, as
|
||||||
line of stdout shares a line with the display. When they are not, the display
|
one terminal is, no line of stdout shares a line with the display. When they
|
||||||
keeps being drawn while stdout waits on a slow reader (2026-10-16,
|
are not, the display keeps being drawn while stdout waits on a slow reader
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/113)
|
(2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113)
|
||||||
|
|
||||||
- Each `scan` phase's progress display names the phase and its place among the
|
- Each `scan` phase's progress display names the phase and its place among the
|
||||||
five, as `[2/5] walk`. It runs from the moment the phase starts and shows:
|
five, as `[2/5] walk`. It runs from the moment the phase starts and shows its
|
||||||
- the progress;
|
progress, elapsed time and ETA. `load` counts the records first. The walk's
|
||||||
- the elapsed time;
|
total is estimated from the records under the operands. The content phase
|
||||||
- the ETA, the time remaining.
|
counts the records it checks from SQLite's grouping alone, then says it is
|
||||||
|
searching the database until the first one is returned. On a terminal a phase
|
||||||
`load` counts the records first. The walk's total is estimated from the
|
with an exact total draws its line followed by the library's bar. Any other
|
||||||
records under the operands. The content phase takes its total from SQLite's
|
phase draws a spinner followed by the line. Every display is drawn again
|
||||||
grouping alone. Until the first record is returned, it says it is searching
|
several times a second. No line reaches the terminal's last column, however
|
||||||
the database. On a terminal a phase with an exact total draws its line
|
narrow, also after the terminal is made narrower during a phase. A line too
|
||||||
followed by the bar of the progress-bar library. Any other phase draws a
|
wide leaves out its rate, then its bar, keeping the phase, progress, elapsed
|
||||||
spinner followed by the line. Every display is drawn again several times a
|
time and ETA whole. It is cut short only when those alone do not fit, as below
|
||||||
second. No line reaches the terminal's last column, however narrow the
|
about 70 columns late in a large scan. When stderr is not a terminal, the line
|
||||||
terminal is, even after it is made narrower during a phase. A line too wide
|
is printed every 5 seconds whether or not an item completed (2026-10-16,
|
||||||
leaves out its rate, then its bar. The rest stays whole. The line is cut
|
|
||||||
short only when the rest alone does not fit, as below about 70 columns late
|
|
||||||
in a large scan. When stderr is not a terminal, the line is printed every 5
|
|
||||||
seconds whether or not an item completed (2026-10-16,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||||
|
|
||||||
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
||||||
they use, as an absolute path. The line says whether that path is the default
|
they use, as an absolute path, and whether that path is the default or comes
|
||||||
or comes from `SFDUPES_DATABASE`. `scan` says when the database is new. README
|
from `SFDUPES_DATABASE`. `scan` says when the database is new. README
|
||||||
"Database" says when `scan` writes each file (2026-10-16,
|
"Database" says when `scan` writes each file (2026-10-16,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/111)
|
https://git.eeqj.de/sneak/sfdupes/issues/111)
|
||||||
|
|
||||||
- The canonical files and model scripts copied again from `sneak/prompts` `next`
|
- The canonical files and model scripts re-vendored from `sneak/prompts` `next`
|
||||||
at `e3f79c4`:
|
at `e3f79c4`:
|
||||||
- `golangci-lint` v2.14.0;
|
- `golangci-lint` v2.14.0;
|
||||||
- lint and test are phases of the `Dockerfile` that write no image, and
|
- lint and test are phases of the `Dockerfile` that write no image, and
|
||||||
`make test` runs the suite under the race detector. Gone are
|
`make test` runs the suite under the race detector. `Dockerfile.lint`,
|
||||||
`Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race`;
|
`script/verify-lint-image-pin` and `make test-race` are gone;
|
||||||
- every `docker build` in `script/` passes a new `CACHEBUST` value. Every
|
- every `docker build` in `script/` passes a new `CACHEBUST` value, which
|
||||||
stage of the `Dockerfile` declares it first, so every `RUN` step runs on
|
every stage of the `Dockerfile` declares first, so every `RUN` step runs
|
||||||
every build. The steps that compile keep Go's build cache, and the tests
|
on every build. The steps that compile keep Go's build cache, and the
|
||||||
run with `-count=1`;
|
tests run with `-count=1`;
|
||||||
- `prettier` runs on the host, from the `node` and `yarn` that
|
- `prettier` runs on the host, from the `node` and `yarn` that
|
||||||
`script/bootstrap` installs. The `prettier` and `markdown` stages are
|
`script/bootstrap` installs, so the `prettier` and `markdown` stages are
|
||||||
gone, and the build stage installs `git` and `make` itself;
|
gone and the build stage installs `git` and `make` itself;
|
||||||
- a new push cancels the workflow's older run on the same branch, and a run
|
- a new push cancels the workflow's older run on the same branch, and a run
|
||||||
stops after 20 minutes;
|
stops after 20 minutes;
|
||||||
- a stray editor-settings file is deleted.
|
- `.claude/settings.json` is deleted.
|
||||||
|
|
||||||
Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before
|
Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before
|
||||||
`ARG CACHEBUST`:
|
`ARG CACHEBUST`: the module download in `lint`, `test` and the build stage,
|
||||||
- the module download in `lint`, `test` and the build stage;
|
and the install of `git` and `make` in the build stage. Docker keeps them
|
||||||
- the install of `git` and `make` in the build stage.
|
between builds, and every step that lints, tests or compiles still runs on
|
||||||
|
every build (2026-10-15, https://git.eeqj.de/sneak/sfdupes/issues/95)
|
||||||
Docker keeps them between builds. Every step that checks or compiles the
|
|
||||||
code still runs on every build (2026-10-15,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95)
|
|
||||||
|
|
||||||
- With `-x`, an operand under another operand is no longer dropped when the
|
- With `-x`, an operand under another operand is no longer dropped when the
|
||||||
outer operand's walk cannot reach it. That is the case when a directory
|
outer operand's walk cannot reach it. That is the case when a directory
|
||||||
@@ -279,8 +235,8 @@ are known.
|
|||||||
https://git.eeqj.de/sneak/sfdupes/issues/106)
|
https://git.eeqj.de/sneak/sfdupes/issues/106)
|
||||||
|
|
||||||
- With `-x`, `scan` keeps the records below a directory on another filesystem
|
- With `-x`, `scan` keeps the records below a directory on another filesystem
|
||||||
under its operand, instead of deleting them. It counts their sizes in the size
|
under its operand, and counts their sizes in the size census, instead of
|
||||||
census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
|
deleting them (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
|
||||||
|
|
||||||
- The hash and content phases open each file without blocking and check that it
|
- The hash and content phases open each file without blocking and check that it
|
||||||
is still a regular file. A file replaced by a FIFO or device node after it was
|
is still a regular file. A file replaced by a FIFO or device node after it was
|
||||||
@@ -310,12 +266,12 @@ are known.
|
|||||||
- `make test-race` ran the test suite under the race detector in a cgo-enabled
|
- `make test-race` ran the test suite under the race detector in a cgo-enabled
|
||||||
container, outside `make check`. Since
|
container, outside `make check`. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
|
https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
|
||||||
under the race detector, in the `Dockerfile`'s Debian-based `test` phase.
|
under the race detector, in the `Dockerfile`'s Debian-based `test` phase, and
|
||||||
`make test-race` is gone (2026-10-04,
|
`make test-race` is gone (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/18)
|
https://git.eeqj.de/sneak/sfdupes/issues/18)
|
||||||
|
|
||||||
- A bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
|
- A bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
|
||||||
rather than serve the gates, the lint and test steps, from cache. Since
|
rather than serve the gates from cache. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
|
https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
|
||||||
`REPO_POLICIES.md` requires, and may serve the gates from cache. The builds in
|
`REPO_POLICIES.md` requires, and may serve the gates from cache. The builds in
|
||||||
`script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04,
|
`script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04,
|
||||||
@@ -328,14 +284,9 @@ are known.
|
|||||||
- `script/lint` writes no image, so a run no longer leaves an untagged one
|
- `script/lint` writes no image, so a run no longer leaves an untagged one
|
||||||
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
|
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
|
||||||
|
|
||||||
- Tests cover:
|
- Tests cover a missing database, `scan` keeping stdout empty, its skip warning,
|
||||||
- a missing database;
|
the `report` and `trees` summary lines, and every subcommand going through
|
||||||
- `scan` keeping stdout empty;
|
`runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
||||||
- its skip warning;
|
|
||||||
- the `report` and `trees` summary lines;
|
|
||||||
- every subcommand going through `runE`.
|
|
||||||
|
|
||||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
|
|
||||||
|
|
||||||
- `.golangci.yml` replaced with the current canonical copy, which uses
|
- `.golangci.yml` replaced with the current canonical copy, which uses
|
||||||
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
||||||
@@ -373,7 +324,7 @@ are known.
|
|||||||
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
|
||||||
and copied the sources with `--chown`, so no `chown -R` walked them. Since
|
and copied the sources with `--chown`, so no `chown -R` walked them. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
|
https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
|
||||||
nothing changes owner. The build stage only compiles, as root, and the tests
|
nothing changes owner: the build stage only compiles, as root, and the tests
|
||||||
run as `nobody` in the `test` phase (2026-10-04,
|
run as `nobody` in the `test` phase (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/43)
|
https://git.eeqj.de/sneak/sfdupes/issues/43)
|
||||||
|
|
||||||
@@ -388,19 +339,13 @@ are known.
|
|||||||
The schema gains the `files_signature` index (2026-10-04,
|
The schema gains the `files_signature` index (2026-10-04,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/14)
|
https://git.eeqj.de/sneak/sfdupes/issues/14)
|
||||||
|
|
||||||
- Progress prints at once on a non-terminal and uses a real terminal test.
|
- Progress prints at once on a non-terminal, uses a real terminal test, and
|
||||||
Warnings print through a spinner instead of racing its redraw (2026-10-03,
|
prints warnings through a spinner instead of racing its redraw (2026-10-03,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/13)
|
https://git.eeqj.de/sneak/sfdupes/issues/13)
|
||||||
|
|
||||||
- These operands are warned about and skipped, and the records beneath them are
|
- Symlink, socket, FIFO, device and `.zfs` operands are warned about and
|
||||||
kept:
|
skipped, and the records beneath them are kept (2026-10-03,
|
||||||
- symlinks;
|
https://git.eeqj.de/sneak/sfdupes/issues/9)
|
||||||
- sockets;
|
|
||||||
- FIFOs;
|
|
||||||
- device nodes;
|
|
||||||
- `.zfs` directories.
|
|
||||||
|
|
||||||
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/9)
|
|
||||||
|
|
||||||
- `scan` holds a lock on a lock file beside the database for its whole run, so a
|
- `scan` holds a lock on a lock file beside the database for its whole run, so a
|
||||||
second `scan` fails at once with exit 1 (2026-10-03,
|
second `scan` fails at once with exit 1 (2026-10-03,
|
||||||
@@ -414,13 +359,8 @@ are known.
|
|||||||
WAL mode, so reading needs only read access (2026-10-03,
|
WAL mode, so reading needs only read access (2026-10-03,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/8)
|
https://git.eeqj.de/sneak/sfdupes/issues/8)
|
||||||
|
|
||||||
- Four characters are escaped in `report`, `trees` and warning paths:
|
- Tabs, newlines, carriage returns and backslashes are escaped in report, trees
|
||||||
- tab;
|
and warning paths. The root directory's path is `/` (2026-10-03,
|
||||||
- newline;
|
|
||||||
- carriage return;
|
|
||||||
- backslash.
|
|
||||||
|
|
||||||
The root directory's path is `/` (2026-10-03,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
https://git.eeqj.de/sneak/sfdupes/issues/7)
|
||||||
|
|
||||||
- A plain `docker build .` stamps the `git` tag or short commit instead of
|
- A plain `docker build .` stamps the `git` tag or short commit instead of
|
||||||
@@ -435,14 +375,14 @@ are known.
|
|||||||
size, and README "Duplicate detection" documented every size threshold. A file
|
size, and README "Duplicate detection" documented every size threshold. A file
|
||||||
under 10 MiB was hashed in full, and its `head`, `tail` and `content` all held
|
under 10 MiB was hashed in full, and its `head`, `tail` and `content` all held
|
||||||
that hash. A larger file got only its 64 KiB `head` and `tail` in the hash
|
that hash. A larger file got only its 64 KiB `head` and `tail` in the hash
|
||||||
phase. The content phase ran after the update phase. It read a file for
|
phase. The content phase, after the update phase, read it for `content` only
|
||||||
`content` only when its size, `head` and `tail` matched another record's, from
|
when its size, `head` and `tail` matched another record's, from this scan or
|
||||||
this scan or an earlier one. `content` was the whole file below 50 MiB, and
|
an earlier one. `content` was the whole file below 50 MiB, and gigabyte-spaced
|
||||||
gigabyte-spaced 1 MiB samples at or above. A file gone or changed since its
|
1 MiB samples at or above. A file gone or changed since its record was written
|
||||||
record was written was never read. `report` and `trees` left out any record
|
was never read. `report` and `trees` left out any record without a `content`
|
||||||
without a `content` hash. The `content` column is part of the version 1
|
hash. The `content` column is part of the version 1 schema. Superseded by
|
||||||
schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132
|
https://git.eeqj.de/sneak/sfdupes/issues/132 (2026-09-22, branch `next`,
|
||||||
(2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61)
|
https://git.eeqj.de/sneak/sfdupes/issues/61)
|
||||||
|
|
||||||
- The dead `files.dat` references removed from `Makefile`, `.gitignore` and
|
- The dead `files.dat` references removed from `Makefile`, `.gitignore` and
|
||||||
`.dockerignore` (2026-09-21, branch `next`,
|
`.dockerignore` (2026-09-21, branch `next`,
|
||||||
@@ -453,8 +393,8 @@ are known.
|
|||||||
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
|
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
|
||||||
without the false `(Debian-based)` note or the tag. Since
|
without the false `(Debian-based)` note or the tag. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
|
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
|
||||||
holds the only `golangci-lint` pin, in that form. `Dockerfile.lint` is gone,
|
holds the only `golangci-lint` pin, in that form. `Dockerfile.lint` and
|
||||||
and so is `script/verify-lint-image-pin`, which compared the two pins
|
`script/verify-lint-image-pin`, which compared the two pins, are gone
|
||||||
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/25)
|
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/25)
|
||||||
|
|
||||||
- All linting ran in Docker, via `Dockerfile.lint` and `script/lint`. Per the
|
- All linting ran in Docker, via `Dockerfile.lint` and `script/lint`. Per the
|
||||||
@@ -463,8 +403,8 @@ are known.
|
|||||||
`script/bootstrap` stopped installing or pinning the linter, and
|
`script/bootstrap` stopped installing or pinning the linter, and
|
||||||
`script/verify-linter-pin` was retired. Since
|
`script/verify-linter-pin` was retired. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
|
https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
|
||||||
`Dockerfile`'s `lint` phase. `script/lint` builds that phase alone, and the
|
`Dockerfile`'s `lint` phase, which `script/lint` builds alone and the build
|
||||||
build stage depends on it through `COPY --from=lint /src/go.sum /dev/null`.
|
stage depends on through `COPY --from=lint /src/go.sum /dev/null`.
|
||||||
`Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
|
`Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
|
||||||
an image build may run `docker`, so the phase calls `golangci-lint` directly
|
an image build may run `docker`, so the phase calls `golangci-lint` directly
|
||||||
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/46)
|
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/46)
|
||||||
@@ -473,43 +413,38 @@ are known.
|
|||||||
`script/bootstrap` instead of `apk add --no-cache make` inline.
|
`script/bootstrap` instead of `apk add --no-cache make` inline.
|
||||||
`script/verify-linter-pin` failed the build unless the linter copied from the
|
`script/verify-linter-pin` failed the build unless the linter copied from the
|
||||||
lint stage was the version `script/bootstrap` pinned. Builds then took up to
|
lint stage was the version `script/bootstrap` pinned. Builds then took up to
|
||||||
5m14s from an empty cache, mostly in a `chown -R` of the module cache, filed
|
5m14s cold, mostly in a `chown -R` of the module cache, filed as
|
||||||
as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
|
https://git.eeqj.de/sneak/sfdupes/issues/43. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
|
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
|
||||||
`make` with `apk add --no-cache` and only compiles. The `lint` and `test`
|
`make` with `apk add --no-cache` and only compiles; the `lint` and `test`
|
||||||
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
|
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/42)
|
https://git.eeqj.de/sneak/sfdupes/issues/42)
|
||||||
|
|
||||||
- The Docker layer cache is defeated for the gate steps, so `script/cibuild` and
|
- The Docker layer cache is busted for the gate steps, so `script/cibuild` and
|
||||||
`script/docker` cannot report a pass they did not earn. The `Dockerfile`
|
`script/docker` cannot report a green they did not earn. The `Dockerfile`
|
||||||
copies the tree before its gates. On an unchanged tree Docker served them from
|
copies the tree before its gates, so on an unchanged tree Docker served them
|
||||||
cache, and the build exited 0 without running them. The fix was a
|
from cache and the build exited 0 having run nothing. The fix was a
|
||||||
`CHECK_EPOCH` build argument. Since
|
`CHECK_EPOCH` build argument. Since
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
|
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
|
||||||
passes a new `CACHEBUST` value instead. Every stage of the `Dockerfile`
|
passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile`
|
||||||
declares it before every step that checks or compiles the code. Run as root,
|
declares before every step that lints, tests or compiles. Run as root, the
|
||||||
the tests fail `TestScanHardlinkRunFailsTogether`, because root reads through
|
tests fail `TestScanHardlinkRunFailsTogether`, because root reads through the
|
||||||
the `chmod(0)` the test relies on. So the `test` phase runs them as `nobody`
|
`chmod(0)` the test relies on, so the `test` phase runs them as `nobody`
|
||||||
(2026-08-09, branch `cibuild-cache-bust`,
|
(2026-08-09, branch `cibuild-cache-bust`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/32)
|
https://git.eeqj.de/sneak/sfdupes/issues/32)
|
||||||
|
|
||||||
- `script/bootstrap` checked the installed `golangci-lint` version instead of
|
- `script/bootstrap` checked the installed `golangci-lint` version instead of
|
||||||
only its presence. The version lived only in `GOLANGCI_LINT_VERSION`, with the
|
only its presence. The version lived only in `GOLANGCI_LINT_VERSION`, with the
|
||||||
`go install` module reference derived from it. Any installed version that was
|
`go install` module ref derived from it. Any installed version that was not
|
||||||
not the pin was reinstalled, whether:
|
the pin was reinstalled: older, newer, absent or unparseable. `go install`
|
||||||
- older;
|
writes into `GOBIN` (or `GOPATH/bin`) while `make lint` ran the first
|
||||||
- newer;
|
`golangci-lint` on `PATH`, so bootstrap re-read the effective version after
|
||||||
- absent;
|
installing. On a mismatch it printed both paths and both versions and exited
|
||||||
- unparseable.
|
non-zero; it did not reorder `PATH` or delete anyone's binary. The `--version`
|
||||||
|
call kept its stderr and was bounded by `timeout(1)` where that exists. `git`,
|
||||||
`go install` writes into `GOBIN`, or `GOPATH/bin`, while `make lint` ran the
|
`make` and `go` kept presence-only checks. The version check was verified by
|
||||||
first `golangci-lint` on `PATH`. So bootstrap re-read the effective version
|
bootstrapping this host from v2.10.1 to v2.12.2, and again to a no-op. Stub
|
||||||
after installing. On a mismatch it printed both paths and both versions and
|
runs of the script under `dash` covered:
|
||||||
exited non-zero. It did not reorder `PATH` or delete anyone's binary. The
|
|
||||||
`--version` call kept its stderr and was bounded by `timeout(1)` where that
|
|
||||||
exists. `git`, `make` and `go` kept presence-only checks. The version check
|
|
||||||
was verified by bootstrapping this host from v2.10.1 to v2.12.2, and again
|
|
||||||
to a no-op. Stub runs of the script under `dash` covered:
|
|
||||||
- a thirteen-input version-parse matrix;
|
- a thirteen-input version-parse matrix;
|
||||||
- a shadowed install, which must exit non-zero;
|
- a shadowed install, which must exit non-zero;
|
||||||
- an install destination not on `PATH`;
|
- an install destination not on `PATH`;
|
||||||
@@ -518,7 +453,7 @@ are known.
|
|||||||
|
|
||||||
`make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been
|
`make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been
|
||||||
hiding findings on `main`. Superseded by
|
hiding findings on `main`. Superseded by
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/46, since the linter is no longer
|
https://git.eeqj.de/sneak/sfdupes/issues/46: the linter is no longer
|
||||||
installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
installed on a host (2026-08-09, branch `bootstrap-version-check`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
https://git.eeqj.de/sneak/sfdupes/issues/24)
|
||||||
|
|
||||||
@@ -526,66 +461,45 @@ are known.
|
|||||||
context-aware `hashPool`. Every blocking send in the feeder and the workers
|
context-aware `hashPool`. Every blocking send in the feeder and the workers
|
||||||
selects on `ctx.Done()`, and `jobs` is closed on every path out. `hashPhase`
|
selects on `ctx.Done()`, and `jobs` is closed on every path out. `hashPhase`
|
||||||
defers `pool.stop()`, which cancels and then drains `results` until the last
|
defers `pool.stop()`, which cancels and then drains `results` until the last
|
||||||
goroutine has exited. Draining is what frees a worker parked on a send. `ctx`
|
goroutine has exited; draining is what frees a worker parked on a send. `ctx`
|
||||||
is passed from `cmd.Context()` as the first parameter, through:
|
is threaded from `cmd.Context()` through `runScan`, `syncScan`, both worker
|
||||||
- `runScan`;
|
pools and the whole database layer, as the first parameter everywhere. The
|
||||||
- `syncScan`;
|
walk pool gets the same treatment, plus a `ctx.Err()` guard after the walk. A
|
||||||
- both worker pools;
|
cancelled walk yields a partial size census, and every file it never reached
|
||||||
- the whole database layer.
|
looks vanished to the update phase. That phase's own `BeginTx` also fails on
|
||||||
|
the cancelled context before deleting anything. The guard is still the barrier
|
||||||
The walk pool gets the same treatment, plus a `ctx.Err()` guard after the
|
once an interrupted scan may commit what it has. Tests drive `run(scan)`
|
||||||
walk. A cancelled walk yields a partial size census, and every file it never
|
against a database whose insert trigger aborts, and assert that the scan fails
|
||||||
reached looks vanished to the update phase. That phase's own `BeginTx` also
|
instead of hanging and that the goroutine count returns to its pre-scan
|
||||||
fails on the cancelled context before deleting anything. The guard is still
|
baseline. Other tests cancel a scan part-way through the walk,
|
||||||
the barrier once an interrupted scan may commit what it has. Tests drive
|
|
||||||
`run(scan)` against a database whose insert trigger aborts. They assert that
|
|
||||||
the scan fails instead of hanging, and that the goroutine count returns to
|
|
||||||
its pre-scan baseline. Other tests cancel a scan part-way through the walk,
|
|
||||||
deterministically, by counting the scan's own checks of `ctx.Done()`. They
|
deterministically, by counting the scan's own checks of `ctx.Done()`. They
|
||||||
assert that it stops at the guard with every record intact. It then holds a
|
assert that it stops at the guard, holding a partial size census and a
|
||||||
partial size census and a still-populated record index. Direct tests cover
|
still-populated record index, with every record intact. Direct tests of
|
||||||
the remaining cancellation branches of both pools. They test:
|
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
|
||||||
- `sendEvent`;
|
and `hashPhase` cover the remaining cancellation branches of both pools
|
||||||
- the walk workers;
|
|
||||||
- `dispatchDirs`;
|
|
||||||
- `feedHashJobs`;
|
|
||||||
- `hashWorker`;
|
|
||||||
- `hashPhase`.
|
|
||||||
|
|
||||||
(2026-08-09, branch `hash-pool-cleanup`,
|
(2026-08-09, branch `hash-pool-cleanup`,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/6)
|
https://git.eeqj.de/sneak/sfdupes/issues/6)
|
||||||
|
|
||||||
- The database is closed on every fatal exit path. `fatalf` and its `os.Exit(1)`
|
- The database is closed on every fatal exit path. `fatalf` and its `os.Exit(1)`
|
||||||
are gone. So the deferred `db.Close()` runs when a subcommand fails, and with
|
are gone, so the deferred `db.Close()`, and with it the SQLite WAL checkpoint,
|
||||||
it the SQLite WAL checkpoint. These return errors instead:
|
runs when a subcommand fails. `runScan`, `runReport`, `runTrees`,
|
||||||
- `runScan`;
|
`loadRecords` and `resolveRoots` return errors instead. The single exit point
|
||||||
- `runReport`;
|
is `run` in `main.go`. It maps a `fatalError`, anything a subcommand returned,
|
||||||
- `runTrees`;
|
to exit 1, and cobra's own argument and flag errors to exit 2. So a runtime
|
||||||
- `loadRecords`;
|
failure is never reported as a usage error, nor printed with the usage text.
|
||||||
- `resolveRoots`.
|
`main_test.go` drives the CLI in-process and asserts the exit codes from
|
||||||
|
README "Error handling" and the stdout/stderr split. It also checks that a
|
||||||
The single exit point is `run` in `main.go`. It maps a `fatalError`,
|
fatal error after the database is open leaves no `-wal` or `-shm` file behind,
|
||||||
anything a subcommand returned, to exit 1, and cobra's own argument and flag
|
for `scan`, `report` or `trees` (2026-08-09, branch `db-close-on-fatal`,
|
||||||
errors to exit 2. So a runtime failure is never reported as a usage error,
|
|
||||||
nor printed with the usage text. `main_test.go` drives the CLI in-process
|
|
||||||
and asserts the exit codes from README "Error handling" and the
|
|
||||||
stdout/stderr split. It also checks that a fatal error after the database is
|
|
||||||
open leaves no `-wal` or `-shm` file behind, for `scan`, `report` or `trees`
|
|
||||||
(2026-08-09, branch `db-close-on-fatal`,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||||
|
|
||||||
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as
|
- `golangci-lint` updated to v2.12.2 with the canonical config, merged as
|
||||||
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
|
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
|
||||||
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
|
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
|
||||||
replaced with the canonical file. Four thresholds now live under
|
replaced with the canonical file. The linter settings (`lll`, `funlen`,
|
||||||
`linters.settings`, as the v2 schema requires, so they are applied:
|
`cyclop`, `dupl` thresholds) now live under `linters.settings`, per the v2
|
||||||
- `lll`;
|
schema, so they are applied. No new lint findings surfaced (2026-08-09, branch
|
||||||
- `funlen`;
|
`golangci-v2.12.2`, https://git.eeqj.de/sneak/sfdupes/issues/3)
|
||||||
- `cyclop`;
|
|
||||||
- `dupl`.
|
|
||||||
|
|
||||||
No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/3)
|
|
||||||
|
|
||||||
- `Makefile` targets converted to scripts-to-rule-them-all `script/`
|
- `Makefile` targets converted to scripts-to-rule-them-all `script/`
|
||||||
entrypoints, like the other managed repos. All 12 entrypoints exist:
|
entrypoints, like the other managed repos. All 12 entrypoints exist:
|
||||||
@@ -596,17 +510,17 @@ are known.
|
|||||||
Every `Makefile` target does nothing but call its script (2026-07-26, commit
|
Every `Makefile` target does nothing but call its script (2026-07-26, commit
|
||||||
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
|
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
|
||||||
|
|
||||||
- The binary is the default `make` target. Plain `make` builds `sfdupes`, where
|
- The binary is the default `make` target: plain `make` builds `sfdupes`, where
|
||||||
it used to run `check` plus `build`. `make build` remains as an alias
|
it used to run `check` plus `build`. `make build` remains as an alias
|
||||||
(2026-07-24, branch `make-default-target`)
|
(2026-07-24, branch `make-default-target`)
|
||||||
|
|
||||||
- Scan-wide phases, concurrent operands and batched updates. All operands seed
|
- Scan-wide phases, concurrent operands and batched updates. All operands seed
|
||||||
the shared walk pool, and every pass runs once over the whole scan. So the
|
the shared walk pool, and every pass runs once over the whole scan, so totals
|
||||||
totals and ETAs cover the whole scan. The per-operand walk, hash and update
|
and ETAs are scan-global. The per-operand walk, hash and update cycles and
|
||||||
cycles and their stderr announcements are gone. The update pass writes in
|
their stderr announcements are gone. The update pass commits in batched
|
||||||
batches, each in its own transaction. The filesystem is authoritative and the
|
transactions. The filesystem is authoritative and the database an
|
||||||
database follows it, so one scan's changes need not land all at once
|
eventually-consistent reflection of it, so scan-level atomicity is not
|
||||||
(2026-07-24, branch `scan-wide-phases`)
|
required (2026-07-24, branch `scan-wide-phases`)
|
||||||
|
|
||||||
- The stat pass split back out of the walk. Phases are strictly sequential
|
- The stat pass split back out of the walk. Phases are strictly sequential
|
||||||
again, per operand, with parallelism only inside each phase:
|
again, per operand, with parallelism only inside each phase:
|
||||||
@@ -616,8 +530,8 @@ are known.
|
|||||||
- update.
|
- update.
|
||||||
|
|
||||||
The walk enumerates paths with per-directory workers, and the stat pass
|
The walk enumerates paths with per-directory workers, and the stat pass
|
||||||
lstats them with per-file workers. That restores the exact total and ETA of
|
lstats them with per-file workers, restoring the exact total and ETA of the
|
||||||
the stat bar (2026-07-24, branch `parallel-phases`)
|
stat bar (2026-07-24, branch `parallel-phases`)
|
||||||
|
|
||||||
- Each operand is announced on stderr before its passes. With per-operand walk,
|
- Each operand is announced on stderr before its passes. With per-operand walk,
|
||||||
hash and update cycles, a multi-operand run such as `scan /srv/*` showed pass
|
hash and update cycles, a multi-operand run such as `scan /srv/*` showed pass
|
||||||
@@ -625,66 +539,55 @@ are known.
|
|||||||
`scan-operand-progress`)
|
`scan-operand-progress`)
|
||||||
|
|
||||||
- Parallel walk. The walk pass was a single goroutine and took hours at about 20
|
- Parallel walk. The walk pass was a single goroutine and took hours at about 20
|
||||||
million files on a busy pool. One ZFS server took 4 hours for 22 million
|
million files on a busy pool: 22 million files in 4 hours on a ZFS server. It
|
||||||
files. It is now a walk by a pool of per-directory workers that records size
|
is now a per-directory worker-pool traversal that records size and mtime
|
||||||
and mtime as it goes. That folds away the separate stat pass and halves the
|
during the walk, folding away the separate stat pass and halving metadata I/O.
|
||||||
metadata reads. Each `PATH` operand commits in its own transaction, so an
|
Each `PATH` operand commits in its own transaction, so an interrupted scan
|
||||||
interrupted scan keeps completed operands (2026-07-24, branch `parallel-walk`)
|
keeps completed operands (2026-07-24, branch `parallel-walk`)
|
||||||
|
|
||||||
- Persistent scan database. `scan` maintains an SQLite database that survives
|
- Persistent scan database. `scan` maintains a SQLite database
|
||||||
between runs, with the absolute path as each record's key. The driver is
|
(`modernc.org/sqlite`, pure Go, cgo stays disabled), keyed by absolute path,
|
||||||
`modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only new
|
that survives between runs. A rescan hashes only new or changed files, by
|
||||||
or changed files, by mtime and size. It deletes the records of files vanished
|
mtime and size. It deletes the records of files vanished from under the
|
||||||
from under the scanned operands and leaves records outside them untouched. So
|
scanned operands and leaves records outside them untouched, so `scan` can be
|
||||||
`scan` can run daily from `cron`. `report` and `trees` read the database, with
|
cronned daily. `report` and `trees` read the database, with no positional
|
||||||
no positional arguments, instead of a scan stream. The database is at
|
arguments, instead of a scan stream. The database is at
|
||||||
`/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
|
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`. WAL
|
||||||
journaling plus a single-transaction update keep a report run during a scan
|
journaling plus a single-transaction update keep a report run during a scan
|
||||||
safe (2026-07-24, branch `persistent-database`)
|
safe (2026-07-24, branch `persistent-database`)
|
||||||
|
|
||||||
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, added. `v0.0.1`
|
- The `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`) added, `v0.0.1`
|
||||||
tagged, and `main` and the tags pushed (2026-07-23)
|
tagged, and `main` plus tags pushed (2026-07-23)
|
||||||
|
|
||||||
- `scan` command-line rework. Required `PATH...` operands, parsed by cobra,
|
- `scan` CLI rework. Required `PATH...` operands via cobra replace the `/srv`
|
||||||
replace the `/srv` `-root` default. The new `-x`, `--one-file-system` flag
|
`-root` default. The new `-x`/`--one-file-system` flag (GNU convention) stops
|
||||||
follows the GNU convention and stops at filesystem boundaries, which the walk
|
at filesystem boundaries, which are crossed by default (2026-07-23, branch
|
||||||
crosses by default (2026-07-23, branch `scan-required-paths`)
|
`scan-required-paths`)
|
||||||
|
|
||||||
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
|
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
|
||||||
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
|
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
|
||||||
2026-07-22 found these gaps, and the branch closed them:
|
2026-07-22 found these gaps, and the branch closed them:
|
||||||
- `.gitignore`, with the standard patterns for secrets and for OS, editor
|
- `.gitignore`, with OS, editor, Go and secrets patterns; the compiled
|
||||||
and Go files. The compiled `sfdupes` binary and `files.dat` had been
|
`sfdupes` binary and `files.dat` had been sitting untracked in the tree;
|
||||||
sitting untracked in the tree;
|
|
||||||
- `.editorconfig`;
|
- `.editorconfig`;
|
||||||
- `LICENSE` and the README License section. MIT was assumed from house
|
- `LICENSE` and the README License section. MIT was assumed from house
|
||||||
convention, for the owner to confirm;
|
convention, for the owner to confirm;
|
||||||
- `REPO_POLICIES.md` in the repo root;
|
- `REPO_POLICIES.md` in the repo root;
|
||||||
- the canonical `.golangci.yml`, with the 150 findings it raised fixed, so
|
- the canonical `.golangci.yml`, with the 150 findings it raised fixed, so
|
||||||
that `make lint` is clean;
|
that `make lint` is clean;
|
||||||
- the `Makefile` targets the policy requires:
|
- the `Makefile` targets `test`, `lint`, `fmt`, `fmt-check`, `docker` and
|
||||||
- `test` and `lint`;
|
`hooks`. `check` depended on `build`, which wrote the binary, and
|
||||||
- `fmt` and `fmt-check`;
|
`make check` must not modify files, so `check` no longer depends on it;
|
||||||
- `docker` and `hooks`.
|
|
||||||
|
|
||||||
`check` depended on `build`, which wrote the binary. `make check` must
|
|
||||||
not modify files, so `check` no longer depends on `build`;
|
|
||||||
|
|
||||||
- a test suite, which the policy required with a 30-second timeout and the
|
- a test suite, which the policy required with a 30-second timeout and the
|
||||||
conditional `-v` rerun. At 64% coverage, it covers:
|
conditional `-v` rerun. It covers parsing, grouping, digests, suppression,
|
||||||
- parsing and grouping;
|
hashing and the scan pipeline, at 64% coverage;
|
||||||
- digests and suppression;
|
- a multistage `Dockerfile` with hash-pinned images, a fail-fast lint stage
|
||||||
- hashing and the scan pipeline;
|
and a build stage running `make check`, plus `.dockerignore`;
|
||||||
- a multistage `Dockerfile` and `.dockerignore`. The `Dockerfile` had:
|
|
||||||
- hash-pinned images;
|
|
||||||
- a lint stage that failed fast;
|
|
||||||
- a build stage that ran `make check`;
|
|
||||||
- `.gitea/workflows/check.yml`, running `docker build .` on push with the
|
- `.gitea/workflows/check.yml`, running `docker build .` on push with the
|
||||||
checkout action pinned by commit SHA;
|
checkout action pinned by commit SHA;
|
||||||
- the README sections the policy requires. The first is a Description whose
|
- the README sections the policy requires: a Description first line with the
|
||||||
first line names the project, its purpose and category, its license and
|
name, purpose, category, license and author, then Getting Started,
|
||||||
its author. Then come Getting Started and Rationale, and at the end TODO,
|
Rationale, TODO, License and Author;
|
||||||
License and Author;
|
|
||||||
- the stale README non-goal "no git repository setup and no CI" removed;
|
- the stale README non-goal "no git repository setup and no CI" removed;
|
||||||
- the pre-commit hook, installed with `make hooks` once that target existed.
|
- the pre-commit hook, installed with `make hooks` once that target existed.
|
||||||
|
|
||||||
@@ -698,8 +601,8 @@ are known.
|
|||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- Possible later features, out of scope per README, are a byte-for-byte check of
|
- Possible later features, out of scope per README: full-content verification of
|
||||||
candidates and helper scripts for removal.
|
candidates, and helper scripts for removal.
|
||||||
|
|
||||||
# Accepted Divergences
|
# Accepted Divergences
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user