Change detection compares old.mtime < ev.rec.mtime (scan.go:246) where mtime is ModTime().Unix() (scan.go:574, scan.go:713) — whole seconds. A file rewritten in place, at the same size, within the same wall-clock second as the recorded mtime is classified unchanged and keeps its stale hashes forever. No later scan will ever notice.
The code matches the spec: README §Database defines mtime as Unix seconds and §scan mode describes the skip rule in those terms. The problem is that README §scan mode also promises "the database only ever contains signatures verified by the most recent scan that covered them", which reads as a stronger guarantee than the implementation gives.
Two ways to close the gap.
Option A (recommended): document it. Add it to the existing "Known limitation (accepted)" list. Cost: zero. The window requires a same-second, same-size, in-place rewrite, which is rare for the target workload of media, downloads and extracted archives.
Option B: store nanosecond mtime. Eliminates the window permanently, and pre-1.0 is the only free moment to change the schema. But PRAGMA user_version would go to 2, and README §Database makes any unrecognized version fatal — so every existing database is invalidated and every deployment pays a full re-hash of every file. At the stated 150 TB scale that is a very expensive answer to a very rare failure.
Going with Option A. Reopen or comment if you would rather pay for Option B before the tag; after 1.0 it becomes a real migration.
Definition of done
README §scan mode adds the one-second window to the "Known limitation (accepted)" entries, stating the exact conditions (same size, in-place rewrite, within the same second as the recorded mtime) and the consequence (stale hashes retained indefinitely).
The over-strong "only ever contains signatures verified by the most recent scan" sentence is softened so it does not contradict the limitation.
Docs-only change; no code, no schema change.
make check green.
Change detection compares `old.mtime < ev.rec.mtime` (`scan.go:246`) where mtime is `ModTime().Unix()` (`scan.go:574`, `scan.go:713`) — whole seconds. A file rewritten in place, at the same size, within the same wall-clock second as the recorded mtime is classified `unchanged` and keeps its stale hashes forever. No later scan will ever notice.
The code matches the spec: README §Database defines `mtime` as Unix seconds and §scan mode describes the skip rule in those terms. The problem is that README §scan mode also promises "the database only ever contains signatures verified by the most recent scan that covered them", which reads as a stronger guarantee than the implementation gives.
Two ways to close the gap.
**Option A (recommended): document it.** Add it to the existing "Known limitation (accepted)" list. Cost: zero. The window requires a same-second, same-size, in-place rewrite, which is rare for the target workload of media, downloads and extracted archives.
**Option B: store nanosecond mtime.** Eliminates the window permanently, and pre-1.0 is the only free moment to change the schema. But `PRAGMA user_version` would go to 2, and README §Database makes any unrecognized version fatal — so every existing database is invalidated and every deployment pays a full re-hash of every file. At the stated 150 TB scale that is a very expensive answer to a very rare failure.
Going with Option A. Reopen or comment if you would rather pay for Option B before the tag; after 1.0 it becomes a real migration.
## Definition of done
1. README §scan mode adds the one-second window to the "Known limitation (accepted)" entries, stating the exact conditions (same size, in-place rewrite, within the same second as the recorded mtime) and the consequence (stale hashes retained indefinitely).
2. The over-strong "only ever contains signatures verified by the most recent scan" sentence is softened so it does not contradict the limitation.
3. Docs-only change; no code, no schema change.
4. `make check` green.
clawbot
added this to the 1.0.0 milestone 2026-08-09 03:45:25 +02:00
Question for sneak, because 1.0 closes the door on option B: the
database schema is fixed at version 1 once v1.0.0 is tagged, and any
other version is a fatal error with no migration. Storing mtime in
nanoseconds is free today and costs every user a full rescan later.
A (recommended, as the issue says): document the one-second window
as an accepted limitation. Work proceeds on this.
B: store nanosecond mtime now, schema version 2, existing
databases are deleted and rebuilt by the next scan.
If you want B, say so here before the tag and the documentation change
is replaced by the schema change.
Model: fable-5-1
Question for sneak, because 1.0 closes the door on option B: the
database schema is fixed at version 1 once `v1.0.0` is tagged, and any
other version is a fatal error with no migration. Storing mtime in
nanoseconds is free today and costs every user a full rescan later.
- A (recommended, as the issue says): document the one-second window
as an accepted limitation. Work proceeds on this.
- B: store nanosecond mtime now, schema version 2, existing
databases are deleted and rebuilt by the next scan.
If you want B, say so here before the tag and the documentation change
is replaced by the schema change.
Model: fable-5-1
sneak
was assigned by clawbot2026-09-21 09:18:14 +02:00
there are no existing databases. store the timestamps in the highest resolution available. no backward compatibility required before 1.0. nobody has ever used this prototype software.
there are no existing databases. store the timestamps in the highest resolution available. no backward compatibility required before 1.0. nobody has ever used this prototype software.
Ruling (sneak, 2026-10-07, the two comments above): no sfdupes database exists and nobody has used this software, so nothing needs to stay compatible before 1.0. Store the timestamps at the highest resolution available; the schema version does not change. This replaces option A.
Definition of done:
Every timestamp the database stores is kept at the highest resolution the platform gives (for mtime, ModTime().UnixNano()), and change detection compares at that resolution.
PRAGMA user_version stays 1: no migration, no version bump.
A test, failing first, rewrites a file in place at the same size within the same second as its recorded mtime, and shows the next scan re-hashes it.
README §Database and §scan mode state the stored resolution; the one-second window is no longer listed as a known limitation.
Lands on next as one reviewed PR.
Model: opus-5-5
Ruling (sneak, 2026-10-07, the two comments above): no sfdupes database exists and nobody has used this software, so nothing needs to stay compatible before 1.0. Store the timestamps at the highest resolution available; the schema version does not change. This replaces option A.
Definition of done:
- Every timestamp the database stores is kept at the highest resolution the platform gives (for mtime, `ModTime().UnixNano()`), and change detection compares at that resolution.
- `PRAGMA user_version` stays 1: no migration, no version bump.
- A test, failing first, rewrites a file in place at the same size within the same second as its recorded mtime, and shows the next `scan` re-hashes it.
- README §Database and §scan mode state the stored resolution; the one-second window is no longer listed as a known limitation.
- Lands on `next` as one reviewed PR.
Model: opus-5-5
sneak
was unassigned by clawbot2026-10-07 23:11:44 +02:00
clawbot
self-assigned this 2026-10-07 23:11:44 +02:00
Plan. mtime is the only timestamp the database stores.
scan.go: the three ModTime().Unix() calls (the walk record, the record built in the hash phase, and the recheck in unchangedFile) become ModTime().UnixNano(). The change rule stays as it is: a file is unchanged when its size matches and its mtime is not newer than recorded.
The files table keeps mtime INTEGER NOT NULL; PRAGMA user_version stays 1. report and trees never format mtime, so nothing else reads it as seconds.
Test, written and seen failing first: set a file's mtime with os.Chtimes to a time with a fractional second, scan, rewrite it in place at the same size, set its mtime later in that same second, scan again, and check the record now carries the new hashes. Existing tests that compare mtime with Unix() move to UnixNano().
README: the schema comment says Unix nanoseconds, and §scan mode says change detection compares mtime at that resolution. The one-second window was never added to the known limitations, so there is nothing to remove there.
Model: opus-5-5
Plan. `mtime` is the only timestamp the database stores.
- `scan.go`: the three `ModTime().Unix()` calls (the walk record, the record built in the hash phase, and the recheck in `unchangedFile`) become `ModTime().UnixNano()`. The change rule stays as it is: a file is unchanged when its size matches and its mtime is not newer than recorded.
- The `files` table keeps `mtime INTEGER NOT NULL`; `PRAGMA user_version` stays 1. `report` and `trees` never format `mtime`, so nothing else reads it as seconds.
- Test, written and seen failing first: set a file's mtime with `os.Chtimes` to a time with a fractional second, scan, rewrite it in place at the same size, set its mtime later in that same second, scan again, and check the record now carries the new hashes. Existing tests that compare `mtime` with `Unix()` move to `UnixNano()`.
- README: the schema comment says Unix nanoseconds, and §scan mode says change detection compares mtime at that resolution. The one-second window was never added to the known limitations, so there is nothing to remove there.
Model: opus-5-5
clawbot
changed title from Document the one-second mtime change-detection window as a known limitation to Store mtime at nanosecond resolution so a same-second rewrite is re-hashed2026-10-07 23:13:25 +02:00
Built in #96: scan records and compares mtime in nanoseconds, so a same-size rewrite within the same second is re-hashed. The schema version stays 1.
Model: opus-5-5
Built in https://git.eeqj.de/sneak/sfdupes/pulls/96: `scan` records and compares mtime in nanoseconds, so a same-size rewrite within the same second is re-hashed. The schema version stays 1.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Change detection compares
old.mtime < ev.rec.mtime(scan.go:246) where mtime isModTime().Unix()(scan.go:574,scan.go:713) — whole seconds. A file rewritten in place, at the same size, within the same wall-clock second as the recorded mtime is classifiedunchangedand keeps its stale hashes forever. No later scan will ever notice.The code matches the spec: README §Database defines
mtimeas Unix seconds and §scan mode describes the skip rule in those terms. The problem is that README §scan mode also promises "the database only ever contains signatures verified by the most recent scan that covered them", which reads as a stronger guarantee than the implementation gives.Two ways to close the gap.
Option A (recommended): document it. Add it to the existing "Known limitation (accepted)" list. Cost: zero. The window requires a same-second, same-size, in-place rewrite, which is rare for the target workload of media, downloads and extracted archives.
Option B: store nanosecond mtime. Eliminates the window permanently, and pre-1.0 is the only free moment to change the schema. But
PRAGMA user_versionwould go to 2, and README §Database makes any unrecognized version fatal — so every existing database is invalidated and every deployment pays a full re-hash of every file. At the stated 150 TB scale that is a very expensive answer to a very rare failure.Going with Option A. Reopen or comment if you would rather pay for Option B before the tag; after 1.0 it becomes a real migration.
Definition of done
make checkgreen.Question for sneak, because 1.0 closes the door on option B: the
database schema is fixed at version 1 once
v1.0.0is tagged, and anyother version is a fatal error with no migration. Storing mtime in
nanoseconds is free today and costs every user a full rescan later.
as an accepted limitation. Work proceeds on this.
databases are deleted and rebuilt by the next scan.
If you want B, say so here before the tag and the documentation change
is replaced by the schema change.
Model: fable-5-1
there are no existing databases. store the timestamps in the highest resolution available. no backward compatibility required before 1.0. nobody has ever used this prototype software.
schema version does not change
Ruling (sneak, 2026-10-07, the two comments above): no sfdupes database exists and nobody has used this software, so nothing needs to stay compatible before 1.0. Store the timestamps at the highest resolution available; the schema version does not change. This replaces option A.
Definition of done:
ModTime().UnixNano()), and change detection compares at that resolution.PRAGMA user_versionstays 1: no migration, no version bump.scanre-hashes it.nextas one reviewed PR.Model: opus-5-5
Plan.
mtimeis the only timestamp the database stores.scan.go: the threeModTime().Unix()calls (the walk record, the record built in the hash phase, and the recheck inunchangedFile) becomeModTime().UnixNano(). The change rule stays as it is: a file is unchanged when its size matches and its mtime is not newer than recorded.filestable keepsmtime INTEGER NOT NULL;PRAGMA user_versionstays 1.reportandtreesnever formatmtime, so nothing else reads it as seconds.os.Chtimesto a time with a fractional second, scan, rewrite it in place at the same size, set its mtime later in that same second, scan again, and check the record now carries the new hashes. Existing tests that comparemtimewithUnix()move toUnixNano().Model: opus-5-5
Document the one-second mtime change-detection window as a known limitationto Store mtime at nanosecond resolution so a same-second rewrite is re-hashedBuilt in #96:
scanrecords and compares mtime in nanoseconds, so a same-size rewrite within the same second is re-hashed. The schema version stays 1.Model: opus-5-5