From 37cf53e71f8cd596576e0bf585162e744314b660 Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 23 Sep 2026 11:16:58 +0000 Subject: [PATCH] Keep the database schema at version 1 (closes #61) sfdupes is pre-1.0, with no installed base and no databases anywhere, so the schema is changed in place and its version stays 1. schemaVersion goes back to 1; the six-column files table, content included, is the version 1 schema. The check that stops on a database with any other version stays. README.md and TODO.md no longer describe a version 2 or rejecting and rescanning version 1 databases. The main.go package comment still described 1024-byte end windows and said full file contents are never read; it now describes the hashes the code computes. Model: opus-5-5 --- README.md | 7 ++----- TODO.md | 11 +++++------ db.go | 7 ++----- main.go | 15 +++++++++------ 4 files changed, 18 insertions(+), 22 deletions(-) diff --git a/README.md b/README.md index 708ebc6..a36d1a0 100644 --- a/README.md +++ b/README.md @@ -154,11 +154,8 @@ All three subcommands operate on a single SQLite database file: that dies partway leaves a valid database holding everything hashed so far; the next scan skips those records and converges toward the filesystem. -- Schema (`PRAGMA user_version` is the schema version, currently 2; a - database with any other version is a fatal error). Version 2 added - the `content` column and the 64 KiB head/tail signature (replacing - the version 1 1 KiB end windows), so a version 1 database cannot be - reused: it is rejected and the tree must be rescanned from scratch. +- Schema (`PRAGMA user_version` is the schema version, currently 1; a + database with any other version is a fatal error): ```sql CREATE TABLE files ( diff --git a/TODO.md b/TODO.md index 744f174..1199d37 100644 --- a/TODO.md +++ b/TODO.md @@ -36,12 +36,11 @@ `head`, `tail`, and `content` all hold the whole-file hash. A file at 10 MiB or above is gated on the 64 KiB `head` and `tail`, then compared on a `content` hash — the whole file below 50 MiB, - gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2 - (new `content` column); a version 1 database is rejected and must be - rescanned, which is required anyway since every stored hash changed. - `report` and `trees` group by the extended signature, so the ladder is - applied across the whole database. README "Duplicate detection" - documents every rung including the probabilistic large-file path. + gigabyte-spaced 1 MiB samples at or above. The `content` column is + part of the version 1 schema. `report` and `trees` group by the + extended signature, so the ladder is applied across the whole + database. README "Duplicate detection" documents every rung including + the probabilistic large-file path. - remove the dead `files.dat` references from `Makefile`, `.gitignore` and `.dockerignore` (2026-09-21, branch `next`, closes diff --git a/db.go b/db.go index fb1b4dc..8524eff 100644 --- a/db.go +++ b/db.go @@ -25,11 +25,8 @@ const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite" const databaseEnv = "SFDUPES_DATABASE" // schemaVersion is the database schema version this build reads and -// writes, stored in PRAGMA user_version. Version 2 adds the content -// column and the head/tail/content signature (replacing the version 1 -// 1 KiB end windows), so a version 1 database is rejected and must be -// rescanned. -const schemaVersion = 2 +// writes, stored in PRAGMA user_version. +const schemaVersion = 1 // dbDirPerm is the mode for a database parent directory created by // scan. diff --git a/main.go b/main.go index e9012db..1b08d1c 100644 --- a/main.go +++ b/main.go @@ -1,10 +1,13 @@ // Command sfdupes quickly identifies candidate duplicate files across -// very large filesystems without reading full file contents. Files are -// considered duplicates when they have identical size, identical SHA-256 -// of their first 1024 bytes, and identical SHA-256 of their last 1024 -// bytes. scan maintains a persistent SQLite database of file signatures -// (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) that the -// reporting subcommands read. +// very large filesystems without reading every byte of every file. +// Files are considered duplicates when their sizes are equal and they +// agree on a short ladder of SHA-256 hashes. A file under 10 MiB is +// hashed in full. A larger file is compared on the hashes of its first +// and last 64 KiB and on a content hash: of the whole file when it is +// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB +// or larger. scan maintains a persistent SQLite database of file +// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) +// that the reporting subcommands read. // // Usage: //