is there some way to estimate walk time before the walk begins? #161

New Issue

2026-11-11T00:34:03+01:00

sneak commented

2026-11-11 00:34:03 +01:00

No description provided.

clawbot commented

2026-11-11 00:38:09 +01:00

For the worker: this is a question, not a build. Answer it on this issue: can sfdupes estimate the walk time before the walk begins, and how good would the estimate be? Consider what is cheap to know up front (the filesystem's used inode count from statfs, the previous scan's file count and duration in the database) and what is not. Give the options, how accurate each is on a first scan and on later scans, and a recommendation. No code; stop after the comment.

Model: opus-5-5

For the worker: this is a question, not a build. Answer it on this issue: can sfdupes estimate the walk time before the walk begins, and how good would the estimate be? Consider what is cheap to know up front (the filesystem's used inode count from `statfs`, the previous scan's file count and duration in the database) and what is not. Give the options, how accurate each is on a first scan and on later scans, and a recommendation. No code; stop after the comment. Model: opus-5-5

clawbot commented

2026-11-11 00:39:50 +01:00

Short answer: a file count before the walk, yes; a time before the walk, only on a rescan, and only if scan starts recording how long each walk took. On a first scan no cheap source gives a time.

A time needs two numbers: how many files the walk will find, and how fast it finds them. What scan has today:

Rescan: the walk shows ~N files, where N is the number of records the last scan left under the operands. The time remaining appears once about 5 seconds of walking give a rate. Nothing about earlier walk speed is stored.
First scan: no total, so the time remaining reads unknown for the whole walk.

Options:

A. Used-inode count from statfs, first scan only. One cheap call per filesystem gives the number of inodes in use. It is a rough file count, not a time; the time still comes from the measured rate after a few seconds. Where it goes wrong:
- It counts the whole filesystem. scan /srv/photos on a filesystem mounted at /srv gets the count for all of /srv. It fits only when each operand is a mount point, such as / under sfdupes dwim, summed over the filesystems mounted below it.
- It counts directories and symlinks; the walk counts only regular files. Too high, often by a tenth or more.
- A file with several hard links is one inode but several walked paths. A backup tree made of hard links makes the count several times too low.
- btrfs reports zero. NFS reports the server's whole export.
B. Store each walk's duration and file count. The next scan multiplies the stored rate by its own record count and shows a time from the first second. Accuracy depends on whether the kernel still holds the directory listings in memory: a walk soon after another runs many times faster than a cold one. A daily cron scan at the same hour gives a close figure; a rescan an hour after the first does not. It needs one small table in the database.
C. Count files in a separate pass first. Listing every directory is the walk itself, so it costs about as much as the walk. Rejected.

Recommendation: B, if a time at the start of a rescan is wanted; the rate-based time after a few seconds already covers most of the need. Skip A: on a first scan it is right only for mount-point operands without hard-link trees, and it cannot give a time. Both are enhancements and await sneak's approval; nothing is built.

Model: opus-5-5

Short answer: a file count before the walk, yes; a time before the walk, only on a rescan, and only if `scan` starts recording how long each walk took. On a first scan no cheap source gives a time. A time needs two numbers: how many files the walk will find, and how fast it finds them. What `scan` has today: - Rescan: the walk shows `~N` files, where `N` is the number of records the last scan left under the operands. The time remaining appears once about 5 seconds of walking give a rate. Nothing about earlier walk speed is stored. - First scan: no total, so the time remaining reads `unknown` for the whole walk. Options: - **A. Used-inode count from `statfs`, first scan only.** One cheap call per filesystem gives the number of inodes in use. It is a rough file count, not a time; the time still comes from the measured rate after a few seconds. Where it goes wrong: - It counts the whole filesystem. `scan /srv/photos` on a filesystem mounted at `/srv` gets the count for all of `/srv`. It fits only when each operand is a mount point, such as `/` under `sfdupes dwim`, summed over the filesystems mounted below it. - It counts directories and symlinks; the walk counts only regular files. Too high, often by a tenth or more. - A file with several hard links is one inode but several walked paths. A backup tree made of hard links makes the count several times too low. - btrfs reports zero. NFS reports the server's whole export. - **B. Store each walk's duration and file count.** The next scan multiplies the stored rate by its own record count and shows a time from the first second. Accuracy depends on whether the kernel still holds the directory listings in memory: a walk soon after another runs many times faster than a cold one. A daily `cron` scan at the same hour gives a close figure; a rescan an hour after the first does not. It needs one small table in the database. - **C. Count files in a separate pass first.** Listing every directory is the walk itself, so it costs about as much as the walk. Rejected. Recommendation: B, if a time at the start of a rescan is wanted; the rate-based time after a few seconds already covers most of the need. Skip A: on a first scan it is right only for mount-point operands without hard-link trees, and it cannot give a time. Both are enhancements and await sneak's approval; nothing is built. Model: opus-5-5

clawbot referenced this issue

2026-11-11 01:07:59 +01:00

Acceptance test: use sfdupes end to end as a user, and list missing basics #164

clawbot added the needs-decision label 2026-11-11 14:10:29 +01:00

clawbot commented

2026-11-11 14:10:32 +01:00

No PR: this issue is a question, already answered above; storing walk durations (option B) is an enhancement that waits on sneak's approval.

Model: opus-5-5

No PR: this issue is a question, already answered above; storing walk durations (option B) is an enhancement that waits on sneak's approval. Model: opus-5-5

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/sfdupes#161