scan skips OS-appropriate default excludes (/proc, /sys, /tmp, /run, /var/tmp on Linux) #124
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: sneak/sfdupes#124
Reference in New Issue
Block a user
sneak, 2026-10-16, in chat: add OS-appropriate default excludes, so one can run
sfdupes scan /and it skips things like/proc,/sys,/tmp,/runand/var/tmpon Linux.Done when:
sfdupes scan /skips at least/proc,/sys,/dev,/run,/tmpand/var/tmpby default, and also any directory that is the mount point of a kernel or memory-backed filesystem (proc,sysfs,devtmpfs,tmpfs,cgroupand the like) wherever it is mounted./dev,/private/tmp,/private/var/tmp,/private/var/vm,/System/Volumes/VM), and the same file is never counted twice through the/System/Volumes/Datafirmlink.--helpand the README; scan says on stderr which default excludes it applied; one flag turns them off; the user's own excludes still add to them.Model: opus-5-5
Plan for the implementer (one PR on
next, packagemain):sfdupes scan /tmp/xworks (and so do the tests, which run in temp directories). Records below a skipped directory count as outside the scanned operands, as they do for a directory-xdoes not enter: kept, with their sizes in the size census. Skipping by rule is not a warning and not counted as skipped./proc,/sys,/dev,/run,/tmp,/var/tmp, plus every mount point whose filesystem type is kernel or memory-backed, read once from/proc/self/mountinfoas the scan starts:proc,sysfs,devtmpfs,devpts,tmpfs,ramfs,cgroup,cgroup2,debugfs,tracefs,securityfs,pstore,bpf,configfs,fusectl,mqueue,hugetlbfs,efivarfs,binfmt_misc,autofs,nsfs,rpc_pipefs,selinuxfs./dev,/private/tmp,/private/var/tmp,/private/var/vm,/System/Volumes/VM, and/System/Volumes/Data, whose contents a walk of/already reaches through the firmlinks; scanning/System/Volumes/Dataitself still works by the rule above.--no-default-excludesturns the defaults off. No user exclude option is added (sneak asked only for the defaults).scan: skipped /proc (default exclude, proc filesystem)), printed the way warnings are, so it never lands on the progress line.--helplists the defaults for the system it runs on; README lists both systems' defaults and the rule.exclude.go, holds both lists behind a function that takes the system's name, so both lists are tested on Linux, plus themountinforeading; the walk asks it whether to enter a directory./proc/self/mountinfogives/procasproc; a scan of a temp tree whose operand lies inside a default-excluded directory, and a scan with--no-default-excludes.Model: opus-5-5
Correction to the definition of done above: drop "the user's own excludes still add to them". sfdupes has no exclude option of its own, and sneak asked only for OS default excludes, so no
--excludeoption is added here. The rest stands.Model: opus-5-5
Built as planned above in #127; its description lists the two judgement calls (an unreadable
/proc/self/mountinfofails the scan, and where filesystems are mounted over one another the last mount decides).Model: opus-5-5
sneak, 2026-10-17:
> by default it should never scan or account for files on tmpfs no matter how they were arrived at.
This replaces part of the plan above (#124 (comment)) for #127:
tmpfsamong them):/proc/self/mountinfoon Linux, the system's list of mounted filesystems on macOS. It notes each one's device ID, read withlstaton the mount point. The walk then never enters a directory, and never records a file, whose device ID is one of those. That holds however it is reached: below an operand, as an operand itself (scan /tmp/xwhen/tmpis tmpfs), or through a bind mount, which keeps the device ID of what it binds.-x. Each skipped operand or directory gets one stderr line naming it and its filesystem type./proc,/sys,/dev,/run,/tmp,/var/tmp, and the macOS list) stay as planned, with the operand rule: an operand at or inside one of those paths is still walked, unless its filesystem is in the list above.--no-default-excludesturns off both.--no-default-excludes.Model: opus-5-5
Ruling (sneak, 2026-10-17, in chat), correcting the reading in the comment above: the default excludes are paths only, and
--no-default-excludesturns off only that path list. Separately, and always, sfdupes collects no data for anything not on a persistent filesystem (tmpfs, proc, sysfs and other kernel or memory-backed filesystems), however it is reached. That is the program's behaviour, not an exclude, and no flag turns it off.Model: opus-5-5
Ruling (sneak, 2026-10-17, in chat): the default path list is platform-specific, so that on Linux, macOS and Windows alike
sfdupes scanrun against the root does the right thing. So this issue also covers Windows: a default list for a drive's root (for example the page, swap and hibernation files,System Volume Information,$Recycle.Binand the temporary folders), documented beside the Linux and macOS lists. If sfdupes does not build for Windows today, say so here with what it would take, rather than leaving Windows out silently.Model: opus-5-5
sfdupes does not build for Windows today. #127 carries the Windows default path list anyway: as data, tested on Linux through the same function as the macOS list, and documented beside the others. So a Windows build would find it ready.
What a Windows build would take, none of it in that PR:
flock(2)throughgolang.org/x/sys/unix. Windows needsLockFileExfromgolang.org/x/sys/windows, in a file built only for Windows.SFDUPES_DATABASEcomes fromunix.Getwd. Windows would useos.Getwd.syscall.Stat_t, which Windows does not have.-x, the hard-link check and the reads in inode order all use them. The Windows equivalents, a volume serial number and a file index, come fromGetFileInformationByHandle, which has to open each file, and that slows the walk.treesbuilds, and the report's escaping assume/as the only separator and one root. Windows paths have drive letters and backslashes./proc/self/mountinfoon Linux and the mount list on macOS. On Windows it would askGetDriveTypefor RAM disks.make checkwould need a Windows cross-build, so that it keeps building.That is a port of its own, not on the 1.0.0 milestone. Say if you want it filed.
Model: opus-5-5