Keep the database schema at version 1 (closes #61) #63
@@ -154,11 +154,8 @@ All three subcommands operate on a single SQLite database file:
|
||||
that dies partway leaves a valid database holding everything
|
||||
hashed so far; the next scan skips those records and converges
|
||||
toward the filesystem.
|
||||
- Schema (`PRAGMA user_version` is the schema version, currently 2; a
|
||||
database with any other version is a fatal error). Version 2 added
|
||||
the `content` column and the 64 KiB head/tail signature (replacing
|
||||
the version 1 1 KiB end windows), so a version 1 database cannot be
|
||||
reused: it is rejected and the tree must be rescanned from scratch.
|
||||
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
|
||||
database with any other version is a fatal error):
|
||||
|
||||
```sql
|
||||
CREATE TABLE files (
|
||||
|
||||
@@ -36,12 +36,11 @@
|
||||
`head`, `tail`, and `content` all hold the whole-file hash. A file at
|
||||
10 MiB or above is gated on the 64 KiB `head` and `tail`, then
|
||||
compared on a `content` hash — the whole file below 50 MiB,
|
||||
gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
|
||||
(new `content` column); a version 1 database is rejected and must be
|
||||
rescanned, which is required anyway since every stored hash changed.
|
||||
`report` and `trees` group by the extended signature, so the ladder is
|
||||
applied across the whole database. README "Duplicate detection"
|
||||
documents every rung including the probabilistic large-file path.
|
||||
gigabyte-spaced 1 MiB samples at or above. The `content` column is
|
||||
part of the version 1 schema. `report` and `trees` group by the
|
||||
extended signature, so the ladder is applied across the whole
|
||||
database. README "Duplicate detection" documents every rung including
|
||||
the probabilistic large-file path.
|
||||
|
||||
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
|
||||
and `.dockerignore` (2026-09-21, branch `next`, closes
|
||||
|
||||
@@ -25,11 +25,8 @@ const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
|
||||
const databaseEnv = "SFDUPES_DATABASE"
|
||||
|
||||
// schemaVersion is the database schema version this build reads and
|
||||
// writes, stored in PRAGMA user_version. Version 2 adds the content
|
||||
// column and the head/tail/content signature (replacing the version 1
|
||||
// 1 KiB end windows), so a version 1 database is rejected and must be
|
||||
// rescanned.
|
||||
const schemaVersion = 2
|
||||
// writes, stored in PRAGMA user_version.
|
||||
const schemaVersion = 1
|
||||
|
||||
// dbDirPerm is the mode for a database parent directory created by
|
||||
// scan.
|
||||
|
||||
@@ -1,10 +1,13 @@
|
||||
// Command sfdupes quickly identifies candidate duplicate files across
|
||||
// very large filesystems without reading full file contents. Files are
|
||||
// considered duplicates when they have identical size, identical SHA-256
|
||||
// of their first 1024 bytes, and identical SHA-256 of their last 1024
|
||||
// bytes. scan maintains a persistent SQLite database of file signatures
|
||||
// (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) that the
|
||||
// reporting subcommands read.
|
||||
// very large filesystems without reading every byte of every file.
|
||||
// Files are considered duplicates when their sizes are equal and they
|
||||
// agree on a short ladder of SHA-256 hashes. A file under 10 MiB is
|
||||
// hashed in full. A larger file is compared on the hashes of its first
|
||||
// and last 64 KiB and on a content hash: of the whole file when it is
|
||||
// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
|
||||
// or larger. scan maintains a persistent SQLite database of file
|
||||
// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
|
||||
// that the reporting subcommands read.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
|
||||
Reference in New Issue
Block a user