Compare commits

..
Author SHA1 Message Date
sneak 0a4db58fb4 README: guard the daily cron scan against an unmounted disk (closes #175)
check / check (push) Canceled after 0s
A disk that is not mounted leaves its mount point an empty directory.
scan then deletes every record under it, and the next scan reads every
file again. The README now says so, and its cron line runs scan only
when mountpoint reports /srv as mounted.

Model: opus-5-5
2026-11-11 04:56:15 +00:00
clawbot a27d651532 scan: drop the stray [flags] from the usage line (closes #169) (#187)
check / check (push) Canceled after 0s
2026-11-11 05:37:23 +01:00
clawbot a1ec1e0ca4 Hash a file again when its mtime moved back, not only forward (closes #176) (#182)
check / check (push) Canceled after 0s
2026-11-11 05:22:54 +01:00
4 changed files with 61 additions and 23 deletions

No files matched your search

+16 -12
View File
@@ -112,15 +112,20 @@ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
Run `scan` as root, so that it can read every file. `scan` warns about a Run `scan` as root, so that it can read every file. `scan` warns about a
directory it cannot read, and about a new or changed file it cannot read. It directory it cannot read, and about a new or changed file it cannot read. It
skips the path and deletes the records at or below it (see "Rules for the skips the path and deletes the records at or below it (see "Rules for the
walk"). As the file `/etc/cron.d/sfdupes`: walk").
An unmounted disk leaves its mount point empty, so `scan` deletes every record
under it, and the next scan reads every file on the disk again. For each other
disk mounted below `/srv`, add `&& mountpoint -q` and its mount point after the
first test. As the file `/etc/cron.d/sfdupes`:
``` ```
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log 30 3 * * * root if mountpoint -q /srv; then /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log; else echo "a disk is not mounted; scan of /srv skipped"; fi
``` ```
- The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep - The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep
the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` before the the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` directly
command on the same line. before `/usr/local/bin/sfdupes`.
- `scan` writes nothing to stdout. Its stderr, appended here to - `scan` writes nothing to stdout. Its stderr, appended here to
`/var/log/sfdupes.log`, holds: `/var/log/sfdupes.log`, holds:
- one line naming the database; - one line naming the database;
@@ -208,8 +213,8 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
of reads, at the price of a comparison that is not exact. of reads, at the price of a comparison that is not exact.
The hashes live in a database that is kept between runs. A rescan reads only new The hashes live in a database that is kept between runs. A rescan reads only new
files, and files whose size changed or whose modification time (mtime) is newer files, and files whose size or modification time (mtime) differs from their
than recorded. All analysis then works from the database alone. record. All analysis then works from the database alone.
The goal is whole duplicated trees, not only single files. A download extracted The goal is whole duplicated trees, not only single files. A download extracted
twice, or a project copied once too often, is such a tree. It can be removed as twice, or a project copied once too often, is such a tree. It can be removed as
@@ -410,17 +415,16 @@ operands:
- A file not yet in the database is hashed and inserted. - A file not yet in the database is hashed and inserted.
- A file already in the database is not read again when its size and mtime match - A file already in the database is not read again when its size and mtime match
its record. The mtime matches when it is not newer than the recorded one. This its record. This is what makes a daily rescan cheap.
is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks up to seconds after unchanged. ZFS, for example, counts a new file's blocks up to seconds after
the write. A block is the unit in which a filesystem allocates disk space. A the write. A block is the unit in which a filesystem allocates disk space. A
file scanned within those seconds gets its real `disk_size` at the next scan. file scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The - `scan` hashes a file again when its size or its mtime differs from its record.
mtime is compared to the nanosecond, as finely as the filesystem records it. A An older mtime counts as much as a newer one. A file put back by `cp -p`,
same-size rewrite counts as a change whenever the filesystem gives it a later `rsync -a` or a restore from backup often has an older mtime. The mtime is
mtime, even within the same second. compared to the nanosecond, as finely as the filesystem records it.
- The scan deletes each record under a scanned operand whose file it could not - The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without this run. To stat a file is to read its details, such as its size, without
+3
View File
@@ -29,6 +29,9 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- `README.md` says what a disk that is not mounted does to the database. The
daily `cron` line runs `scan` only when `mountpoint` reports `/srv` as mounted
(2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/175)
- The usage line of `scan` no longer ends in `[flags]`. It is the line - The usage line of `scan` no longer ends in `[flags]`. It is the line
`README.md` shows under "Subcommands" (2026-11-11, `README.md` shows under "Subcommands" (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/169) https://git.eeqj.de/sneak/sfdupes/issues/169)
+1 -11
View File
@@ -545,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
prog.increment() prog.increment()
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) { if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
s.toHash = append(s.toHash, ev.rec) s.toHash = append(s.toHash, ev.rec)
continue continue
@@ -780,16 +780,6 @@ func (s *scanState) updatePhase(ctx context.Context) error {
return applyChanges(ctx, s.db, nil, deletes, prog) return applyChanges(ctx, s.db, nil, deletes, prog)
} }
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under // underAnyRoot reports whether path is any of the roots or lies under
// one of them. // one of them.
func underAnyRoot(path string, roots []string) bool { func underAnyRoot(path string, roots []string) bool {
+41
View File
@@ -1384,6 +1384,47 @@ func TestSyncScanMtimeBump(t *testing.T) {
} }
} }
// TestSyncScanMtimeMovedBack replaces a file with other content of the
// same size and an older mtime, as cp -p or a restore from backup does.
// The next scan must hash the file again.
func TestSyncScanMtimeMovedBack(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err := os.Chtimes(a, recorded, recorded)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
replaced := pattern(3, 500)
writeFile(t, dir, "a.bin", replaced)
older := recorded.Add(-time.Hour)
err = os.Chtimes(a, older, older)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
assertWholeFileHashed(t, db, a, replaced)
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
t.Fatalf("mtime = %v, want %v", r.mtime, older)
}
}
// assertWholeFileHashed fails unless the record for path holds the // assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content. // whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string, func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,