sneak/sfdupes
1
0 Fork 0
Code Issues 5 Pull Requests 1 Actions 16 Packages Projects Releases Wiki Activity

proposal: a way to drop the records of a tree that no longer exists #173

New Issue
Closed
opened 2026-11-11 01:06:01 +01:00 by clawbot · 4 comments
clawbot commented 2026-11-11 01:06:01 +01:00
Collaborator
Copy link
Copy source

Proposal from the acceptance test (#164). It needs your decision before anyone builds it.

What is missing

There is no way to tell sfdupes that a tree is gone. After a user deletes a duplicate tree that the report named, its records stay in the database, and report and trees keep listing it.

Example from the test:

  • sfdupes scan tree/elsewhere recorded 20 files, all duplicates of files in other directories.
  • The directory was deleted, as a user acting on the report would do.
  • sfdupes scan tree/elsewhere then failed with sfdupes: resolve .../tree/elsewhere: lstat .../tree/elsewhere: no such file or directory and exit 1. README.md specifies that: an operand that does not exist is a fatal error.
  • report still named the deleted directory in 40 rows, and trees in 2.

The two ways out today:

  • Scan a parent directory that still exists. That walks everything under the parent, which is all of /srv to forget /srv/old-backup.
  • Delete the database and hash everything again.

A cron line with several operands, sfdupes scan /srv/a /srv/b, has the same problem: once /srv/b is removed, every scan fails before it walks /srv/a.

Why a user expects it

Find, delete, look again is what a duplicate finder is used for. The step after deleting should be quick and should not need a walk of the whole disk.

Your choices

  • A subcommand that drops records: for example sfdupes forget PATH... deletes the records at or below each path. It reads no filesystem and takes the scan lock. The name is yours to choose.
  • scan drops a missing operand's records: a missing operand stops being a fatal error. Its records are deleted and the other operands are scanned. The catch: today the fatal error keeps the records safe when an operand is missing only because its disk is not mounted, and that protection would go.
  • Leave as is: the README says to scan the parent.

Recommended: A subcommand that drops records. It is explicit, and it leaves the fatal error, and the protection it gives, in place.

Model: fable-5-2

Proposal from the acceptance test (https://git.eeqj.de/sneak/sfdupes/issues/164). It needs your decision before anyone builds it. ## What is missing There is no way to tell `sfdupes` that a tree is gone. After a user deletes a duplicate tree that the report named, its records stay in the database, and `report` and `trees` keep listing it. Example from the test: - `sfdupes scan tree/elsewhere` recorded 20 files, all duplicates of files in other directories. - The directory was deleted, as a user acting on the report would do. - `sfdupes scan tree/elsewhere` then failed with `sfdupes: resolve .../tree/elsewhere: lstat .../tree/elsewhere: no such file or directory` and exit 1. `README.md` specifies that: an operand that does not exist is a fatal error. - `report` still named the deleted directory in 40 rows, and `trees` in 2. The two ways out today: - Scan a parent directory that still exists. That walks everything under the parent, which is all of `/srv` to forget `/srv/old-backup`. - Delete the database and hash everything again. A `cron` line with several operands, `sfdupes scan /srv/a /srv/b`, has the same problem: once `/srv/b` is removed, every scan fails before it walks `/srv/a`. ## Why a user expects it Find, delete, look again is what a duplicate finder is used for. The step after deleting should be quick and should not need a walk of the whole disk. ## Your choices - **A subcommand that drops records**: for example `sfdupes forget PATH...` deletes the records at or below each path. It reads no filesystem and takes the scan lock. The name is yours to choose. - **`scan` drops a missing operand's records**: a missing operand stops being a fatal error. Its records are deleted and the other operands are scanned. The catch: today the fatal error keeps the records safe when an operand is missing only because its disk is not mounted, and that protection would go. - **Leave as is**: the README says to scan the parent. Recommended: **A subcommand that drops records**. It is explicit, and it leaves the fatal error, and the protection it gives, in place. Model: fable-5-2
sneak was assigned by clawbot 2026-11-11 01:06:01 +01:00
sneak commented 2026-11-11 03:21:26 +01:00
Owner
Copy link
Copy source

no, missing operands are still a fatal error. make the report filter on extant duplicates only when reporting.

no, missing operands are still a fatal error. make the report filter on extant duplicates only when reporting.
sneak commented 2026-11-11 03:22:22 +01:00
Owner
Copy link
Copy source

only have 'report' do stat() on the top level duplicate, not on every file in the proposed tree. it's the user's fault if the report isn't fresh bc they didn't run scan recently enough.

only have 'report' do stat() on the top level duplicate, not on every file in the proposed tree. it's the user's fault if the report isn't fresh bc they didn't run scan recently enough.
clawbot commented 2026-11-11 03:24:03 +01:00
Author
Collaborator
Copy link
Copy source

sneak rejected the proposal and ruled instead (11 November, about 02:21 UTC), which is the definition of done:

  • A missing operand stays a fatal error for scan.
  • report and trees show only duplicates that still exist: for each duplicate group, stat() only its top-level entry (the file, or the root of a duplicate tree), never every file below it, and leave the group out if that entry is gone.
  • A stale report from an old scan is the user's to fix by scanning again; say so in the README.
  • Tests cover a removed top-level duplicate disappearing from the report.

Model: opus-5-5

sneak rejected the proposal and ruled instead (11 November, about 02:21 UTC), which is the definition of done: - A missing operand stays a fatal error for `scan`. - `report` and `trees` show only duplicates that still exist: for each duplicate group, `stat()` only its top-level entry (the file, or the root of a duplicate tree), never every file below it, and leave the group out if that entry is gone. - A stale report from an old scan is the user's to fix by scanning again; say so in the README. - Tests cover a removed top-level duplicate disappearing from the report. Model: opus-5-5
sneak was unassigned by clawbot 2026-11-11 03:24:03 +01:00
clawbot commented 2026-11-11 03:37:00 +01:00
Author
Collaborator
Copy link
Copy source

Built in #184. report checks each path of a possible duplicate group with lstat. trees checks only the directory at the top of each copy. A copy that is gone is left out, and so is a group left with one copy. scan still fails on a missing operand. A permission error counts as present, so an unprivileged user's report stays whole.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/sfdupes/pulls/184. `report` checks each path of a possible duplicate group with `lstat`. `trees` checks only the directory at the top of each copy. A copy that is gone is left out, and so is a group left with one copy. `scan` still fails on a missing operand. A permission error counts as present, so an unprivileged user's report stays whole. Model: opus-5-5
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
critical
Extremely important: data loss, a security hole, production down, or basic use impossible
 merge-ready
Review passed; ready to merge
 needs-checks
CI checks failing or not yet green
 needs-decision
needs-decision
Waits on a decision only sneak can make
 needs-decision
Waits on a decision before work continues
 needs-rebase
Branch conflicts with main; rebase required
 needs-review
PR awaiting independent adversarial review
 needs-rework
Review found blocking problems; author must rework
 post-1.0
postponed by sneak until after 1.0
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/sfdupes#173