Keep the database schema at version 1 (closes #61)
check / check (push) Successful in 55s

sfdupes is pre-1.0, with no installed base and no databases anywhere,
so the schema is changed in place and its version stays 1.
schemaVersion goes back to 1; the six-column files table, content
included, is the version 1 schema. The check that stops on a database
with any other version stays. README.md and TODO.md no longer describe
a version 2 or rejecting and rescanning version 1 databases. The
main.go package comment still described 1024-byte end windows and said
full file contents are never read; it now describes the hashes the
code computes.

Model: opus-5-5
This commit is contained in:
2026-09-23 11:16:58 +00:00
parent 29a65016d0
commit 37cf53e71f
4 changed files with 18 additions and 22 deletions
+2 -5
View File
@@ -154,11 +154,8 @@ All three subcommands operate on a single SQLite database file:
that dies partway leaves a valid database holding everything that dies partway leaves a valid database holding everything
hashed so far; the next scan skips those records and converges hashed so far; the next scan skips those records and converges
toward the filesystem. toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 2; a - Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error). Version 2 added database with any other version is a fatal error):
the `content` column and the 64 KiB head/tail signature (replacing
the version 1 1 KiB end windows), so a version 1 database cannot be
reused: it is rejected and the tree must be rescanned from scratch.
```sql ```sql
CREATE TABLE files ( CREATE TABLE files (
+5 -6
View File
@@ -36,12 +36,11 @@
`head`, `tail`, and `content` all hold the whole-file hash. A file at `head`, `tail`, and `content` all hold the whole-file hash. A file at
10 MiB or above is gated on the 64 KiB `head` and `tail`, then 10 MiB or above is gated on the 64 KiB `head` and `tail`, then
compared on a `content` hash — the whole file below 50 MiB, compared on a `content` hash — the whole file below 50 MiB,
gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2 gigabyte-spaced 1 MiB samples at or above. The `content` column is
(new `content` column); a version 1 database is rejected and must be part of the version 1 schema. `report` and `trees` group by the
rescanned, which is required anyway since every stored hash changed. extended signature, so the ladder is applied across the whole
`report` and `trees` group by the extended signature, so the ladder is database. README "Duplicate detection" documents every rung including
applied across the whole database. README "Duplicate detection" the probabilistic large-file path.
documents every rung including the probabilistic large-file path.
- remove the dead `files.dat` references from `Makefile`, `.gitignore` - remove the dead `files.dat` references from `Makefile`, `.gitignore`
and `.dockerignore` (2026-09-21, branch `next`, closes and `.dockerignore` (2026-09-21, branch `next`, closes
+2 -5
View File
@@ -25,11 +25,8 @@ const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
const databaseEnv = "SFDUPES_DATABASE" const databaseEnv = "SFDUPES_DATABASE"
// schemaVersion is the database schema version this build reads and // schemaVersion is the database schema version this build reads and
// writes, stored in PRAGMA user_version. Version 2 adds the content // writes, stored in PRAGMA user_version.
// column and the head/tail/content signature (replacing the version 1 const schemaVersion = 1
// 1 KiB end windows), so a version 1 database is rejected and must be
// rescanned.
const schemaVersion = 2
// dbDirPerm is the mode for a database parent directory created by // dbDirPerm is the mode for a database parent directory created by
// scan. // scan.
+9 -6
View File
@@ -1,10 +1,13 @@
// Command sfdupes quickly identifies candidate duplicate files across // Command sfdupes quickly identifies candidate duplicate files across
// very large filesystems without reading full file contents. Files are // very large filesystems without reading every byte of every file.
// considered duplicates when they have identical size, identical SHA-256 // Files are considered duplicates when their sizes are equal and they
// of their first 1024 bytes, and identical SHA-256 of their last 1024 // agree on a short ladder of SHA-256 hashes. A file under 10 MiB is
// bytes. scan maintains a persistent SQLite database of file signatures // hashed in full. A larger file is compared on the hashes of its first
// (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) that the // and last 64 KiB and on a content hash: of the whole file when it is
// reporting subcommands read. // under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
// or larger. scan maintains a persistent SQLite database of file
// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
// that the reporting subcommands read.
// //
// Usage: // Usage:
// //