Everything else in the 1.0.0 milestone can be closed by work in this
repo. This cannot: the tool's stated purpose is a sweep of roughly ten
million files and 150 TB on a ZFS server, and TODO.md records no run
at that scale since the 2026-07-24 parallel-walk change, which was
itself prompted by one. The test suite and the README smoke test use trees
of a dozen files. Memory use of report and trees
(#14), non-TTY progress under
cron (#13) and interrupt
handling (#5) are exactly the
things that only show up there.
Only the owner has the hardware, so this is his to run.
Definition of done
With every other issue in the milestone closed and next merged,
a build of main runs on the real server: a first full scan,
a second scan the next day, then report and trees.
Recorded on this issue: wall-clock time of each scan, peak memory
of scan, report and trees, the summary lines, and anything
that looked wrong or hung.
Every defect found is filed as its own issue in the milestone.
v1.0.0 is tagged only after this issue closes.
Model: fable-5-1
Everything else in the `1.0.0` milestone can be closed by work in this
repo. This cannot: the tool's stated purpose is a sweep of roughly ten
million files and 150 TB on a ZFS server, and `TODO.md` records no run
at that scale since the 2026-07-24 parallel-walk change, which was
itself prompted by one. The test suite and the README smoke test use trees
of a dozen files. Memory use of `report` and `trees`
(https://git.eeqj.de/sneak/sfdupes/issues/14), non-TTY progress under
cron (https://git.eeqj.de/sneak/sfdupes/issues/13) and interrupt
handling (https://git.eeqj.de/sneak/sfdupes/issues/5) are exactly the
things that only show up there.
Only the owner has the hardware, so this is his to run.
## Definition of done
1. With every other issue in the milestone closed and `next` merged,
a build of `main` runs on the real server: a first full `scan`,
a second `scan` the next day, then `report` and `trees`.
2. Recorded on this issue: wall-clock time of each scan, peak memory
of `scan`, `report` and `trees`, the summary lines, and anything
that looked wrong or hung.
3. Every defect found is filed as its own issue in the milestone.
4. `v1.0.0` is tagged only after this issue closes.
Model: fable-5-1
clawbot
added this to the 1.0.0 milestone 2026-09-21 09:17:55 +02:00
Assigned to sneak: only you have the server. Nothing here blocks other
work; the rest of the milestone proceeds on next. When the milestone
PR is marked merge-ready, this run is what stands between the merge
and the v1.0.0 tag.
Model: fable-5-1
Assigned to sneak: only you have the server. Nothing here blocks other
work; the rest of the milestone proceeds on `next`. When the milestone
PR is marked `merge-ready`, this run is what stands between the merge
and the `v1.0.0` tag.
Model: fable-5-1
sneak
was assigned by clawbot2026-09-21 09:18:17 +02:00
sfdupes is ready for your trial run. The nine defects that made reports untrustworthy or broke the cron setup are fixed on next, and next passes script/cibuild. The milestone PR #69 lists them and is merge-ready; run from main after you merge it, or from next before.
What changed for the run:
The report and trees TSV escape tab, newline, carriage return and backslash in paths, so a file name cannot break or forge a row.
report and trees need only read access to the database, so you can run them as yourself against root's cron database, and they no longer hold every record in memory.
A truncated report (full disk, closed stdout) exits 1 instead of 0.
Under cron, progress prints a line as each phase starts, so a slow pool no longer looks hung.
A second scan while one is running fails at once instead of doubling the load.
A symlink or other non-directory operand is warned about and skipped; it no longer deletes the records beneath it.
Ctrl-C or SIGTERM saves what was hashed, deletes nothing, and exits 1; the next scan carries on.
README.md now has install steps, a crontab line for the daily scan, and how to read and check the reports.
For the record on this issue (item 2): wall-clock time of each scan, peak memory of scan, report and trees, the summary lines, and anything that looked wrong or hung. The other open milestone issues are build, test and tidy-up work and do not affect the run.
Model: opus-5-5
sfdupes is ready for your trial run. The nine defects that made reports untrustworthy or broke the cron setup are fixed on `next`, and `next` passes `script/cibuild`. The milestone PR https://git.eeqj.de/sneak/sfdupes/pulls/69 lists them and is `merge-ready`; run from `main` after you merge it, or from `next` before.
What changed for the run:
- The report and trees TSV escape tab, newline, carriage return and backslash in paths, so a file name cannot break or forge a row.
- `report` and `trees` need only read access to the database, so you can run them as yourself against root's cron database, and they no longer hold every record in memory.
- A truncated report (full disk, closed stdout) exits 1 instead of 0.
- Under cron, progress prints a line as each phase starts, so a slow pool no longer looks hung.
- A second `scan` while one is running fails at once instead of doubling the load.
- A symlink or other non-directory operand is warned about and skipped; it no longer deletes the records beneath it.
- Ctrl-C or SIGTERM saves what was hashed, deletes nothing, and exits 1; the next scan carries on.
- `README.md` now has install steps, a crontab line for the daily scan, and how to read and check the reports.
For the record on this issue (item 2): wall-clock time of each scan, peak memory of `scan`, `report` and `trees`, the summary lines, and anything that looked wrong or hung. The other open milestone issues are build, test and tidy-up work and do not affect the run.
Model: opus-5-5
Trial under way (sneak, 2026-10-15, on las1stor1): sfdupes scan /srv/ showed the walk at 1,217,446 files after 33.5 minutes, 593 files/s, default --workers. He asked whether that is right.
Answer given: it fits the design. The walk is bound by how fast the pool returns directory and file metadata, not by CPU (README, "scan mode", phase 1: the walk "takes hours at tens of millions of files"), so about 10 million files means roughly 4.5 to 5 hours of walk before hashing starts. The walk runs one worker per CPU by default; on a pool with many disks, --workers well above the core count can help (README, "Concurrency"). zpool iostat -v 5 tells which case applies: disks already busy means more workers will not help much; mostly idle disks means a rerun with --workers 64 should be faster. Interrupting during the walk loses the walk (nothing is written before hashing), though a second walk reads metadata ZFS has just cached.
Model: opus-5-5
Trial under way (sneak, 2026-10-15, on `las1stor1`): `sfdupes scan /srv/` showed the walk at 1,217,446 files after 33.5 minutes, 593 files/s, default `--workers`. He asked whether that is right.
Answer given: it fits the design. The walk is bound by how fast the pool returns directory and file metadata, not by CPU (README, "`scan` mode", phase 1: the walk "takes hours at tens of millions of files"), so about 10 million files means roughly 4.5 to 5 hours of walk before hashing starts. The walk runs one worker per CPU by default; on a pool with many disks, `--workers` well above the core count can help (README, "Concurrency"). `zpool iostat -v 5` tells which case applies: disks already busy means more workers will not help much; mostly idle disks means a rerun with `--workers 64` should be faster. Interrupting during the walk loses the walk (nothing is written before hashing), though a second walk reads metadata ZFS has just cached.
Model: opus-5-5
The tool should always give the user some feedback to show it is doing something and isn't hung. It should show what phase it's on, out of how many phases, and show phase progress, elapsed, and eta.
The scan mode walk doesn't tell you how many phases besides walk there are.
```
root@las1stor1:~# go/bin/sfdupes trees
<hangs>
```
The tool should always give the user some feedback to show it is doing something and isn't hung. It should show what phase it's on, out of how many phases, and show phase progress, elapsed, and eta.
The scan mode walk doesn't tell you how many phases besides walk there are.
root@las1stor1:~# go/bin/sfdupes trees
^C
root@las1stor1:~#
Took far too long, strace on the pid was hanging too. No idea why it was hanging, I don't even know where the database file is/was. Please diagnose.
```
root@las1stor1:~# go/bin/sfdupes trees
^C
root@las1stor1:~#
```
Took far too long, strace on the pid was hanging too. No idea why it was hanging, I don't even know where the database file is/was. Please diagnose.
Diagnosis: the sfdupes in ~/go/bin is almost certainly the old v0.0.1 from July, not the release candidate. That explains all three reports.
Why that build.go install sneak.berlin/go/sfdupes@latest picks the newest release tag, and Go does not count v1.0.0-rc1 or v1.0.0-rc2 as releases, so @latest gives v0.0.1. To check: go version -m ~/go/bin/sfdupes | grep -w mod prints the version it was built from.
What v0.0.1 does. It has no database. scan ignores SFDUPES_DATABASE, keeps everything in memory and writes its records to stdout at the end, so nothing appears in /srv/tmp/sfdupes. Its walk uses one thread whatever --workers says. report and trees read that output from a file named on the command line, or from stdin when none is named. So a bare sfdupes trees sat waiting for keyboard input, and strace only showed it blocked reading the terminal. No lock and no disk wait was involved. I reproduced both with a v0.0.1 build here.
What the release candidate does (v1.0.0-rc2; same Go code as main and next):
The database is /var/lib/sfdupes/db.sqlite, or the path in SFDUPES_DATABASE; that variable name is right.
scan creates the directory, the database file and a .lock file beside it as it starts, and -wal and -shm files before the walk. The walk writes no records. Records are first committed during the hash phase, 10,000 at a time, and the rest in the update and content phases.
report and trees stop at once with an error naming the path if there is no database. They never wait on a scan's lock and can read a scan in progress.
Still wrong on the release candidate, getting its own issues and fixes now: report and trees print nothing until they finish (a minute or more at your size); the scan's progress does not say which phase out of how many; nothing says which database file is in use; and a database inside the scanned tree (/srv/tmp/... under /srv/) gets its own files recorded.
What to run now
Ctrl-C the running scan. It writes nothing to disk and would print binary records to the terminal at the end.
go install sneak.berlin/go/sfdupes@v1.0.0-rc2, then check with the go version -m line above.
Put the database outside /srv/, with room for a few GB (about 700 bytes per file here). The default needs only unset SFDUPES_DATABASE. Then run ~/go/bin/sfdupes scan --workers 64 /srv/. Its display goes load, walk, hash, update, content.
From a second shell, ls -la /var/lib/sfdupes/ shows the files from the first second. Once hashing has started, ~/go/bin/sfdupes report > /dev/null prints how many records are in the database so far.
Model: opus-5-5
Diagnosis: the `sfdupes` in `~/go/bin` is almost certainly the old `v0.0.1` from July, not the release candidate. That explains all three reports.
**Why that build.** `go install sneak.berlin/go/sfdupes@latest` picks the newest release tag, and Go does not count `v1.0.0-rc1` or `v1.0.0-rc2` as releases, so `@latest` gives `v0.0.1`. To check: `go version -m ~/go/bin/sfdupes | grep -w mod` prints the version it was built from.
**What `v0.0.1` does.** It has no database. `scan` ignores `SFDUPES_DATABASE`, keeps everything in memory and writes its records to stdout at the end, so nothing appears in `/srv/tmp/sfdupes`. Its walk uses one thread whatever `--workers` says. `report` and `trees` read that output from a file named on the command line, or from stdin when none is named. So a bare `sfdupes trees` sat waiting for keyboard input, and strace only showed it blocked reading the terminal. No lock and no disk wait was involved. I reproduced both with a `v0.0.1` build here.
**What the release candidate does** (`v1.0.0-rc2`; same Go code as `main` and `next`):
- The database is `/var/lib/sfdupes/db.sqlite`, or the path in `SFDUPES_DATABASE`; that variable name is right.
- `scan` creates the directory, the database file and a `.lock` file beside it as it starts, and `-wal` and `-shm` files before the walk. The walk writes no records. Records are first committed during the hash phase, 10,000 at a time, and the rest in the update and content phases.
- `report` and `trees` stop at once with an error naming the path if there is no database. They never wait on a scan's lock and can read a scan in progress.
Still wrong on the release candidate, getting its own issues and fixes now: `report` and `trees` print nothing until they finish (a minute or more at your size); the scan's progress does not say which phase out of how many; nothing says which database file is in use; and a database inside the scanned tree (`/srv/tmp/...` under `/srv/`) gets its own files recorded.
**What to run now**
1. Ctrl-C the running scan. It writes nothing to disk and would print binary records to the terminal at the end.
2. `go install sneak.berlin/go/sfdupes@v1.0.0-rc2`, then check with the `go version -m` line above.
3. Put the database outside `/srv/`, with room for a few GB (about 700 bytes per file here). The default needs only `unset SFDUPES_DATABASE`. Then run `~/go/bin/sfdupes scan --workers 64 /srv/`. Its display goes `load`, `walk`, `hash`, `update`, `content`.
4. From a second shell, `ls -la /var/lib/sfdupes/` shows the files from the first second. Once hashing has started, `~/go/bin/sfdupes report > /dev/null` prints how many records are in the database so far.
Model: opus-5-5
make go install ...@latest install the latest green commit, i.e. main's tip.
make sfdupes info show the database information: size, rows, etc.
root@las1stor1:~# SFDUPES_DATABASE="/srv/tmp/sfdupes/db.db" ./go/bin/sfdupes report
first dupe size
report: 0 records read, 0 duplicate groups, 0 dupe files, 0 B reclaimable
root@las1stor1:~#
seems to be correct.
your todo:
1. make `go install ...@latest` install the latest green commit, i.e. main's tip.
2. make `sfdupes info` show the database information: size, rows, etc.
```
root@las1stor1:~# SFDUPES_DATABASE="/srv/tmp/sfdupes/db.db" ./go/bin/sfdupes report
first dupe size
report: 0 records read, 0 duplicate groups, 0 dupe files, 0 B reclaimable
root@las1stor1:~#
```
Your trial's findings are queued for sfdupes' manager: the duplicate check you specified (#132), raw hash storage (#134) and the display fixes (#133). This comes back to you, with a link to install the build, once they are on next.
Model: opus-5-5
Your trial's findings are queued for sfdupes' manager: the duplicate check you specified (https://git.eeqj.de/sneak/sfdupes/issues/132), raw hash storage (https://git.eeqj.de/sneak/sfdupes/issues/134) and the display fixes (https://git.eeqj.de/sneak/sfdupes/issues/133). This comes back to you, with a link to install the build, once they are on `next`.
Model: opus-5-5
sneak
was unassigned by clawbot2026-10-22 19:53:25 +02:00
clawbot
self-assigned this 2026-10-22 19:53:25 +02:00
Your trial can be rerun. next now has the duplicate check you specified, each hash stored as raw bytes, and the display fixes; #137 lists them. Install it with:
go install sneak.berlin/go/sfdupes@next
Model: opus-5-5
Your trial can be rerun. `next` now has the duplicate check you specified, each hash stored as raw bytes, and the display fixes; https://git.eeqj.de/sneak/sfdupes/pulls/137 lists them. Install it with:
```
go install sneak.berlin/go/sfdupes@next
```
Model: opus-5-5
clawbot
removed their assignment 2026-10-26 06:34:49 +01:00
sneak
was assigned by clawbot2026-10-26 06:34:49 +01:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Everything else in the
1.0.0milestone can be closed by work in thisrepo. This cannot: the tool's stated purpose is a sweep of roughly ten
million files and 150 TB on a ZFS server, and
TODO.mdrecords no runat that scale since the 2026-07-24 parallel-walk change, which was
itself prompted by one. The test suite and the README smoke test use trees
of a dozen files. Memory use of
reportandtrees(#14), non-TTY progress under
cron (#13) and interrupt
handling (#5) are exactly the
things that only show up there.
Only the owner has the hardware, so this is his to run.
Definition of done
nextmerged,a build of
mainruns on the real server: a first fullscan,a second
scanthe next day, thenreportandtrees.of
scan,reportandtrees, the summary lines, and anythingthat looked wrong or hung.
v1.0.0is tagged only after this issue closes.Model: fable-5-1
Assigned to sneak: only you have the server. Nothing here blocks other
work; the rest of the milestone proceeds on
next. When the milestonePR is marked
merge-ready, this run is what stands between the mergeand the
v1.0.0tag.Model: fable-5-1
sfdupes is ready for your trial run. The nine defects that made reports untrustworthy or broke the cron setup are fixed on
next, andnextpassesscript/cibuild. The milestone PR #69 lists them and ismerge-ready; run frommainafter you merge it, or fromnextbefore.What changed for the run:
reportandtreesneed only read access to the database, so you can run them as yourself against root's cron database, and they no longer hold every record in memory.scanwhile one is running fails at once instead of doubling the load.README.mdnow has install steps, a crontab line for the daily scan, and how to read and check the reports.For the record on this issue (item 2): wall-clock time of each scan, peak memory of
scan,reportandtrees, the summary lines, and anything that looked wrong or hung. The other open milestone issues are build, test and tidy-up work and do not affect the run.Model: opus-5-5
Trial under way (sneak, 2026-10-15, on
las1stor1):sfdupes scan /srv/showed the walk at 1,217,446 files after 33.5 minutes, 593 files/s, default--workers. He asked whether that is right.Answer given: it fits the design. The walk is bound by how fast the pool returns directory and file metadata, not by CPU (README, "
scanmode", phase 1: the walk "takes hours at tens of millions of files"), so about 10 million files means roughly 4.5 to 5 hours of walk before hashing starts. The walk runs one worker per CPU by default; on a pool with many disks,--workerswell above the core count can help (README, "Concurrency").zpool iostat -v 5tells which case applies: disks already busy means more workers will not help much; mostly idle disks means a rerun with--workers 64should be faster. Interrupting during the walk loses the walk (nothing is written before hashing), though a second walk reads metadata ZFS has just cached.Model: opus-5-5
The tool should always give the user some feedback to show it is doing something and isn't hung. It should show what phase it's on, out of how many phases, and show phase progress, elapsed, and eta.
The scan mode walk doesn't tell you how many phases besides walk there are.
Took far too long, strace on the pid was hanging too. No idea why it was hanging, I don't even know where the database file is/was. Please diagnose.
root@las1stor1:
# export SFDUPES_DATABASE="/srv/tmp/sfdupes/db.db"# ./go/bin/sfdupes scan --workers 150 /srv/root@las1stor1:
⠼ walk (1250586/-, 11448 files/s) [1m10s]
meanwhile, nothing is created in
/srv/tmp/sfdupes.Diagnosis: the
sfdupesin~/go/binis almost certainly the oldv0.0.1from July, not the release candidate. That explains all three reports.Why that build.
go install sneak.berlin/go/sfdupes@latestpicks the newest release tag, and Go does not countv1.0.0-rc1orv1.0.0-rc2as releases, so@latestgivesv0.0.1. To check:go version -m ~/go/bin/sfdupes | grep -w modprints the version it was built from.What
v0.0.1does. It has no database.scanignoresSFDUPES_DATABASE, keeps everything in memory and writes its records to stdout at the end, so nothing appears in/srv/tmp/sfdupes. Its walk uses one thread whatever--workerssays.reportandtreesread that output from a file named on the command line, or from stdin when none is named. So a baresfdupes treessat waiting for keyboard input, and strace only showed it blocked reading the terminal. No lock and no disk wait was involved. I reproduced both with av0.0.1build here.What the release candidate does (
v1.0.0-rc2; same Go code asmainandnext):/var/lib/sfdupes/db.sqlite, or the path inSFDUPES_DATABASE; that variable name is right.scancreates the directory, the database file and a.lockfile beside it as it starts, and-waland-shmfiles before the walk. The walk writes no records. Records are first committed during the hash phase, 10,000 at a time, and the rest in the update and content phases.reportandtreesstop at once with an error naming the path if there is no database. They never wait on a scan's lock and can read a scan in progress.Still wrong on the release candidate, getting its own issues and fixes now:
reportandtreesprint nothing until they finish (a minute or more at your size); the scan's progress does not say which phase out of how many; nothing says which database file is in use; and a database inside the scanned tree (/srv/tmp/...under/srv/) gets its own files recorded.What to run now
go install sneak.berlin/go/sfdupes@v1.0.0-rc2, then check with thego version -mline above./srv/, with room for a few GB (about 700 bytes per file here). The default needs onlyunset SFDUPES_DATABASE. Then run~/go/bin/sfdupes scan --workers 64 /srv/. Its display goesload,walk,hash,update,content.ls -la /var/lib/sfdupes/shows the files from the first second. Once hashing has started,~/go/bin/sfdupes report > /dev/nullprints how many records are in the database so far.Model: opus-5-5
seems to be correct.
your todo:
make
go install ...@latestinstall the latest green commit, i.e. main's tip.make
sfdupes infoshow the database information: size, rows, etc.clawbot referenced this issue2026-10-16 15:28:32 +02:00
Your trial's findings are queued for sfdupes' manager: the duplicate check you specified (#132), raw hash storage (#134) and the display fixes (#133). This comes back to you, with a link to install the build, once they are on
next.Model: opus-5-5
v0.0.3is tagged onmain's tip: https://git.eeqj.de/sneak/sfdupes/src/tag/v0.0.3Model: opus-5-5
Your trial can be rerun.
nextnow has the duplicate check you specified, each hash stored as raw bytes, and the display fixes; #137 lists them. Install it with:Model: opus-5-5