Keep the database schema at version 1 (closes #61) #63
@@ -154,11 +154,8 @@ All three subcommands operate on a single SQLite database file:
|
|||||||
that dies partway leaves a valid database holding everything
|
that dies partway leaves a valid database holding everything
|
||||||
hashed so far; the next scan skips those records and converges
|
hashed so far; the next scan skips those records and converges
|
||||||
toward the filesystem.
|
toward the filesystem.
|
||||||
- Schema (`PRAGMA user_version` is the schema version, currently 2; a
|
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
|
||||||
database with any other version is a fatal error). Version 2 added
|
database with any other version is a fatal error):
|
||||||
the `content` column and the 64 KiB head/tail signature (replacing
|
|
||||||
the version 1 1 KiB end windows), so a version 1 database cannot be
|
|
||||||
reused: it is rejected and the tree must be rescanned from scratch.
|
|
||||||
|
|
||||||
```sql
|
```sql
|
||||||
CREATE TABLE files (
|
CREATE TABLE files (
|
||||||
|
|||||||
@@ -36,12 +36,11 @@
|
|||||||
`head`, `tail`, and `content` all hold the whole-file hash. A file at
|
`head`, `tail`, and `content` all hold the whole-file hash. A file at
|
||||||
10 MiB or above is gated on the 64 KiB `head` and `tail`, then
|
10 MiB or above is gated on the 64 KiB `head` and `tail`, then
|
||||||
compared on a `content` hash — the whole file below 50 MiB,
|
compared on a `content` hash — the whole file below 50 MiB,
|
||||||
gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
|
gigabyte-spaced 1 MiB samples at or above. The `content` column is
|
||||||
(new `content` column); a version 1 database is rejected and must be
|
part of the version 1 schema. `report` and `trees` group by the
|
||||||
rescanned, which is required anyway since every stored hash changed.
|
extended signature, so the ladder is applied across the whole
|
||||||
`report` and `trees` group by the extended signature, so the ladder is
|
database. README "Duplicate detection" documents every rung including
|
||||||
applied across the whole database. README "Duplicate detection"
|
the probabilistic large-file path.
|
||||||
documents every rung including the probabilistic large-file path.
|
|
||||||
|
|
||||||
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
|
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
|
||||||
and `.dockerignore` (2026-09-21, branch `next`, closes
|
and `.dockerignore` (2026-09-21, branch `next`, closes
|
||||||
|
|||||||
@@ -25,11 +25,8 @@ const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
|
|||||||
const databaseEnv = "SFDUPES_DATABASE"
|
const databaseEnv = "SFDUPES_DATABASE"
|
||||||
|
|
||||||
// schemaVersion is the database schema version this build reads and
|
// schemaVersion is the database schema version this build reads and
|
||||||
// writes, stored in PRAGMA user_version. Version 2 adds the content
|
// writes, stored in PRAGMA user_version.
|
||||||
// column and the head/tail/content signature (replacing the version 1
|
const schemaVersion = 1
|
||||||
// 1 KiB end windows), so a version 1 database is rejected and must be
|
|
||||||
// rescanned.
|
|
||||||
const schemaVersion = 2
|
|
||||||
|
|
||||||
// dbDirPerm is the mode for a database parent directory created by
|
// dbDirPerm is the mode for a database parent directory created by
|
||||||
// scan.
|
// scan.
|
||||||
|
|||||||
@@ -1,10 +1,13 @@
|
|||||||
// Command sfdupes quickly identifies candidate duplicate files across
|
// Command sfdupes quickly identifies candidate duplicate files across
|
||||||
// very large filesystems without reading full file contents. Files are
|
// very large filesystems without reading every byte of every file.
|
||||||
// considered duplicates when they have identical size, identical SHA-256
|
// Files are considered duplicates when their sizes are equal and they
|
||||||
// of their first 1024 bytes, and identical SHA-256 of their last 1024
|
// agree on a short ladder of SHA-256 hashes. A file under 10 MiB is
|
||||||
// bytes. scan maintains a persistent SQLite database of file signatures
|
// hashed in full. A larger file is compared on the hashes of its first
|
||||||
// (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) that the
|
// and last 64 KiB and on a content hash: of the whole file when it is
|
||||||
// reporting subcommands read.
|
// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
|
||||||
|
// or larger. scan maintains a persistent SQLite database of file
|
||||||
|
// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
|
||||||
|
// that the reporting subcommands read.
|
||||||
//
|
//
|
||||||
// Usage:
|
// Usage:
|
||||||
//
|
//
|
||||||
|
|||||||
Reference in New Issue
Block a user