sneak/sfdupes
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions 5 Packages Projects Releases Wiki Activity

Carry the README changes from next into the rewrite
check / check (push) Waiting to run
Details

Browse Source 
`next` changed the old README after this branch was cut, in
#146 (the `totals` table, commit
98193b2) and #147 (`disk_size`,
commit 4ed2e37). This commit states the same facts in the rewritten
README, in the places #140 (comment)
lists, and changes nothing else. Every statement was checked against the
code on `next`.

Model: fable-5-1
This commit is contained in:
sneak
2026-11-02 22:02:50 +00:00
parent d8b58bf281
commit f373b7d9e4
1 changed files with 101 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+101 -47
View File
@@ -405,6 +405,11 @@ operands:
- A file already in the database is not read again when its size and mtime match - A file already in the database is not read again when its size and mtime match
its record. The mtime matches when it is not newer than the recorded one. This its record. The mtime matches when it is not newer than the recorded one. This
is what makes a daily rescan cheap. is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks only seconds after the
write. A block is the unit in which a filesystem allocates disk space. A file
scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The - `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A mtime is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later same-size rewrite counts as a change whenever the filesystem gives it a later
@@ -435,15 +440,17 @@ operands:
phases. phases.
1. `load` reads the existing records. Each record under the operands is kept in 1. `load` reads the existing records. Each record under the operands is kept in
memory by path, with its size and mtime, for the walk to compare against. memory by path, with its size, its size on disk and its mtime, for the walk
to compare against.
2. `walk` lists the trees under all operands. Several workers share one queue of 2. `walk` lists the trees under all operands. Several workers share one queue of
directories to read. A worker is one thread of the program. A thread is one directories to read. A worker is one thread of the program. A thread is one
of several parts of a program that run at the same time and share its memory. of several parts of a program that run at the same time and share its memory.
"Concurrency" says how many workers there are. Each worker reads one "Concurrency" says how many workers there are. Each worker reads one
directory at a time. It hands the subdirectories it finds back to the queue, directory at a time. It hands the subdirectories it finds back to the queue,
and it stats each regular file as it finds it. The walk compares each file and it stats each regular file as it finds it. The walk compares each file
with its record. An unchanged file needs nothing more. Every new or changed with its record. An unchanged file is not read again. When its size on disk
file is passed to the hash phase. has changed, the update phase writes its new `disk_size`. Every new or
changed file is passed to the hash phase.
3. `hash` reads every new or changed file and computes its three hashes. Files 3. `hash` reads every new or changed file and computes its three hashes. Files
are read in inode order. An inode is the filesystem's own record of a file. are read in inode order. An inode is the filesystem's own record of a file.
Files with nearby inode numbers usually lie near each other on disk. Reading Files with nearby inode numbers usually lie near each other on disk. Reading
@@ -452,8 +459,9 @@ phases.
tree made of hard links therefore costs one read per file. Completed records tree made of hard links therefore costs one read per file. Completed records
are written to the database 10,000 at a time while hashing runs. So a scan are written to the database 10,000 at a time while hashing runs. So a scan
interrupted after hours keeps everything hashed so far. interrupted after hours keeps everything hashed so far.
4. `update` writes the hashed records not yet written, then deletes the records 4. `update` writes the hashed records not yet written, and the new `disk_size`
the scan did not verify. of each unchanged file whose size on disk changed. Then it deletes the
records the scan did not verify.
Reading a tree one directory at a time waits on the disk at every step and takes Reading a tree one directory at a time waits on the disk at every step and takes
hours at millions of files. Reading many directories at once is what makes the hours at millions of files. Reading many directories at once is what makes the
@@ -656,14 +664,14 @@ Processing:
the same for the same database contents. the same for the same database contents.
`report` has two phases, each with its own progress display on stderr (see `report` has two phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `read` counts the records, then reads "Progress"). `open` opens the database. `read` gets the number of records from
them in signature order, largest size first. The database keeps an index named the `totals` table (see "Schema"), then reads them in signature order, largest
`files_signature`, a sorted list of the records by signature (see "Schema"). size first. The database keeps an index named `files_signature`, a sorted list
Reading it backwards gives that order, so SQLite sorts nothing and returns the of the records by signature (see "Schema"). Reading it backwards gives that
first record at once. Records with one signature come together, and so do the order, so SQLite sorts nothing and returns the first record at once. Records
records of one size. So `report` holds only the duplicate groups of the size it with one signature come together, and so do the records of one size. So `report`
is reading. Once it reaches a smaller size, it writes them, ordered by `first` holds only the duplicate groups of the size it is reading. Once it reaches a
path. smaller size, it writes them, ordered by `first` path.
#### Report output format #### Report output format
@@ -746,22 +754,35 @@ Two limitations are accepted:
two trees differ. two trees differ.
`trees` has four phases, each with its own progress display on stderr (see `trees` has four phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `read` counts the records, then reads "Progress"). `open` opens the database. `read` gets the number of records from
them in path order. It builds the directory tree and computes the digests as it the `totals` table, then reads them in path order. It builds the directory tree
goes. `group` groups the directories by digest and picks the groups to report. and computes the digests as it goes. `group` groups the directories by digest
`write` writes them to stdout. and picks the groups to report. `write` writes them to stdout.
#### Trees output format #### Trees output format
The report is TSV on stdout, with a header line and then one row per duplicate The report is TSV on stdout, with a header line and then one row per duplicate
tree. A group of N trees gives N-1 rows. `files` is the number of regular files tree. A group of N trees gives N-1 rows. After the `first` and `dupe` paths come
in one copy of the tree, and `size` their total size in bytes: three numbers:
- `files`, the number of regular files in one copy of the tree;
- `size`, their total size in bytes;
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
added up.
``` ```
first dupe files size first dupe files size disk_size
/srv/a/project /srv/backup/project 3417 104857600 /srv/a/project /srv/backup/project 3417 104857600 61865984
``` ```
A file's `disk_size` is the space its blocks take on disk. It can be more than
the file's size, because space is allocated in whole blocks. It is less than the
file's size on a compressed or sparse file. A sparse file has gaps that take no
space on disk. So two copies of one tree can differ in `disk_size`. When a file
in one copy is a hard link to one in the other, both copies count its blocks in
full. Removing it from one copy then frees nothing, as the limitation under
"`trees` mode" says.
The paths are escaped as described under "Report output format". The root The paths are escaped as described under "Report output format". The root
directory's path is `/`. directory's path is `/`.
@@ -808,11 +829,11 @@ size of recorded files 150000000000000 (136.4 TiB)
of all recorded files, in bytes, then rounded to a unit such as TiB. of all recorded files, in bytes, then rounded to a unit such as TiB.
`info` has two phases, each with its own progress display on stderr (see `info` has two phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `count` reads the schema version and "Progress"). `open` opens the database. `count` reads the schema version and the
counts the records in one transaction. A transaction is a group of reads or row of `totals` (see "Schema") in one transaction. A transaction is a group of
writes that SQLite carries out as one step, against the database as it stands at reads or writes that SQLite carries out as one step, against the database as it
one moment. So both describe the database at one moment, even while a scan is stands at one moment. So both describe the database at one moment, even while a
writing. `info` writes its lines after the `count` display has ended. scan is writing. `info` writes its lines after the `count` display has ended.
### Database ### Database
@@ -908,33 +929,36 @@ Only one `scan` at a time can hold the lock.
- Then, before the walk as well, it writes: - Then, before the walk as well, it writes:
- the `-wal` and `-shm` files, which stay beside the database while the scan - the `-wal` and `-shm` files, which stay beside the database while the scan
runs; runs;
- in a new database, the table, the index and the schema version described - in a new database, the tables, the index, the triggers and the schema
under "Schema". version described under "Schema".
- During the `load` and `walk` phases, no records are written. A first scan's - During the `load` and `walk` phases, no records are written. A first scan's
database holds no records until the hash phase, however long its walk runs. database holds no records until the hash phase, however long its walk runs.
- During the `hash` phase, records are written 10,000 at a time. - During the `hash` phase, records are written 10,000 at a time.
- During the `update` phase, the rest of the hashed records are written, and the - During the `update` phase, the rest of the hashed records are written, and so
deletions are made. is the new `disk_size` of each unchanged file whose size on disk changed. Then
the deletions are made.
- As `scan` closes the database, the `-wal` and `-shm` files go away, unless a - As `scan` closes the database, the `-wal` and `-shm` files go away, unless a
report still has the database open. report still has the database open.
#### Schema #### Schema
The schema is the database's layout, its table and its index. The schema is the database's layout: its tables, its index and its triggers. A
trigger is a statement SQLite runs by itself when a table changes.
`PRAGMA user_version` holds the schema version, currently 1. An empty database `PRAGMA user_version` holds the schema version, currently 1. An empty database
is at version 0, and `scan` sets it up. Any other version is a fatal error. is at version 0, and `scan` sets it up. Any other version is a fatal error.
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal `report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
error too. `scan` creates the table and the index and sets the version in one error too. `scan` creates the tables, the index and the triggers and sets the
transaction (see "`info` mode"). So the three land together or not at all. A version in one transaction (see "`info` mode"). So they all land together or not
first scan stopped while doing so leaves an empty database, which the next scan at all. A first scan stopped while doing so leaves an empty database, which the
sets up. A database at version 0 that already has a `files` table was not made next scan sets up. A database at version 0 that already has a `files` table was
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
the file and rescan. remove the file and rescan.
```sql ```sql
CREATE TABLE files ( CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes size INTEGER NOT NULL, -- bytes
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512)
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
@@ -942,10 +966,28 @@ CREATE TABLE files (
content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
) WITHOUT ROWID; ) WITHOUT ROWID;
CREATE INDEX files_signature ON files (size, head, tail, content); CREATE INDEX files_signature ON files (size, head, tail, content);
CREATE TABLE totals (
records INTEGER NOT NULL, -- number of records in files
size INTEGER NOT NULL -- sizes of all records in files added up
);
INSERT INTO totals (records, size) VALUES (0, 0);
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
UPDATE totals SET records = records + 1, size = size + NEW.size;
END;
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
UPDATE totals SET records = records - 1, size = size - OLD.size;
END;
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
UPDATE totals SET size = size + NEW.size - OLD.size;
END;
``` ```
- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not - `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
necessarily UTF-8 text. necessarily UTF-8 text.
- `disk_size` is the space the file's blocks take on disk. `lstat` is the system
call that stats a file without following a symlink. Its `st_blocks` field
counts that space in units of 512 bytes. `disk_size` is less than `size` on a
compressed or sparse file (see "Trees output format").
- `mtime` holds the whole seconds of the file's mtime, rounded down, and - `mtime` holds the whole seconds of the file's mtime, rounded down, and
`mtime_nsec` the nanoseconds past that second. Split this way, they hold any `mtime_nsec` the nanoseconds past that second. Split this way, they hold any
mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
@@ -956,6 +998,16 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
not 32 bytes. not 32 bytes.
- `files_signature` lets `report` read the records in signature order without - `files_signature` lets `report` read the records in signature order without
sorting the table. sorting the table.
- `totals` holds one row: the number of records in `files` and the sum of their
sizes. Three triggers on `files` keep that row current. Each runs in the
transaction of the change to `files` that fires it, so the row and the records
always agree. `info` reads both numbers from this row. The `load` phase of
`scan` and the `read` phase of `report` and `trees` read the number of records
from it as their total. Counting the records in `files` would read every
record. That takes minutes on a database of tens of millions. The triggers
make each insert, each delete and each change of a record's `size` slower, so
a first scan of many small files takes about two fifths longer. On large
files, reading them takes most of the time, so the difference is small.
### Progress ### Progress
@@ -966,8 +1018,9 @@ of its line are listed below.
The phases, and what each display counts: The phases, and what each display counts:
- `scan` has four phases: - `scan` has four phases:
- `[1/4] load` counts the records first, then reads them against that count. - `[1/4] load` gets the number of records from the `totals` table first,
While it counts, the display says `counting the records`. then reads them against that number. Until it has the number, the display
says `counting the records`.
- `[2/4] walk` has no exact total. Its estimate is the number of records - `[2/4] walk` has no exact total. Its estimate is the number of records
under the operands, shown with a `~`. A scan with no records under its under the operands, shown with a `~`. A scan with no records under its
operands, such as a first scan, has no estimate. The time remaining is operands, such as a first scan, has no estimate. The time remaining is
@@ -979,8 +1032,9 @@ The phases, and what each display counts:
- `report` has two phases: - `report` has two phases:
- `[1/2] open` has no total. The display says `opening the database`, also - `[1/2] open` has no total. The display says `opening the database`, also
while it waits on a busy one. while it waits on a busy one.
- `[2/2] read` counts the records first, then reads them against that count. - `[2/2] read` gets the number of records from the `totals` table first,
It writes rows to stdout as it goes. then reads them against that number. Until it has the number, the display
says `counting the records`. It writes rows to stdout as it goes.
- `trees` has four phases: - `trees` has four phases:
- `[1/4] open` and `[2/4] read` are as for `report`, except that `read` - `[1/4] open` and `[2/4] read` are as for `report`, except that `read`
writes nothing. writes nothing.
@@ -988,9 +1042,8 @@ The phases, and what each display counts:
- `[4/4] write` has the duplicate-tree groups as its total. - `[4/4] write` has the duplicate-tree groups as its total.
- `info` has two phases: - `info` has two phases:
- `[1/2] open` is as for `report`. - `[1/2] open` is as for `report`.
- `[2/2] count` has no total either, because one request to the database - `[2/2] count` has no total either, because it reads one row. The display
counts everything. The display says `counting the records` until that says `counting the records` until it has the row.
request returns.
Each display is one line holding: Each display is one line holding:
@@ -1323,9 +1376,10 @@ All of the following, run in this directory, must pass:
- The groups come in order of size, largest first. - The groups come in order of size, largest first.
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`. `trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
The row shows 2 files and 3100 bytes. The two `sub` directories are left The row shows 2 files, 3100 bytes, and a `disk_size` that depends on the
out, because the `t1` and `t2` group implies them. `t3` appears nowhere, filesystem. The two `sub` directories are left out, because the `t1` and
since one of its file names differs. `t2` group implies them. `t3` appears nowhere, since one of its file names
differs.
After the rescan, the second `report` shows `one.bin` out of its group, After the rescan, the second `report` shows `one.bin` out of its group,
because its content changed. `copy.bin` and `copy2.bin` remain as one pair. because its content changed. `copy.bin` and `copy2.bin` remain as one pair.