sneak/sfdupes
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions 5 Packages Projects Releases Wiki Activity

Carry the README changes from next into the rewrite
check / check (push) Waiting to run
Details

Browse Source 
`next` changed the old README after this branch was cut, in
#146 (the `totals` table, commit
98193b2) and #147 (`disk_size`,
commit 4ed2e37). This commit states the same facts in the rewritten
README, in the places #140 (comment)
lists, and changes nothing else. Every statement was checked against the
code on `next`.

Model: fable-5-1
This commit is contained in:
sneak
2026-11-02 22:02:50 +00:00
parent d8b58bf281
commit f373b7d9e4
1 changed files with 101 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+101 -47
View File
@@ -405,6 +405,11 @@ operands:
- A file already in the database is not read again when its size and mtime match
its record. The mtime matches when it is not newer than the recorded one. This
is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks only seconds after the
write. A block is the unit in which a filesystem allocates disk space. A file
scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later
@@ -435,15 +440,17 @@ operands:
phases.
1. `load` reads the existing records. Each record under the operands is kept in
memory by path, with its size and mtime, for the walk to compare against.
memory by path, with its size, its size on disk and its mtime, for the walk
to compare against.
2. `walk` lists the trees under all operands. Several workers share one queue of
directories to read. A worker is one thread of the program. A thread is one
of several parts of a program that run at the same time and share its memory.
"Concurrency" says how many workers there are. Each worker reads one
directory at a time. It hands the subdirectories it finds back to the queue,
and it stats each regular file as it finds it. The walk compares each file
with its record. An unchanged file needs nothing more. Every new or changed
file is passed to the hash phase.
with its record. An unchanged file is not read again. When its size on disk
has changed, the update phase writes its new `disk_size`. Every new or
changed file is passed to the hash phase.
3. `hash` reads every new or changed file and computes its three hashes. Files
are read in inode order. An inode is the filesystem's own record of a file.
Files with nearby inode numbers usually lie near each other on disk. Reading
@@ -452,8 +459,9 @@ phases.
tree made of hard links therefore costs one read per file. Completed records
are written to the database 10,000 at a time while hashing runs. So a scan
interrupted after hours keeps everything hashed so far.
4. `update` writes the hashed records not yet written, then deletes the records
the scan did not verify.
4. `update` writes the hashed records not yet written, and the new `disk_size`
of each unchanged file whose size on disk changed. Then it deletes the
records the scan did not verify.
Reading a tree one directory at a time waits on the disk at every step and takes
hours at millions of files. Reading many directories at once is what makes the
@@ -656,14 +664,14 @@ Processing:
the same for the same database contents.
`report` has two phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `read` counts the records, then reads
them in signature order, largest size first. The database keeps an index named
`files_signature`, a sorted list of the records by signature (see "Schema").
Reading it backwards gives that order, so SQLite sorts nothing and returns the
first record at once. Records with one signature come together, and so do the
records of one size. So `report` holds only the duplicate groups of the size it
is reading. Once it reaches a smaller size, it writes them, ordered by `first`
path.
"Progress"). `open` opens the database. `read` gets the number of records from
the `totals` table (see "Schema"), then reads them in signature order, largest
size first. The database keeps an index named `files_signature`, a sorted list
of the records by signature (see "Schema"). Reading it backwards gives that
order, so SQLite sorts nothing and returns the first record at once. Records
with one signature come together, and so do the records of one size. So `report`
holds only the duplicate groups of the size it is reading. Once it reaches a
smaller size, it writes them, ordered by `first` path.
#### Report output format
@@ -746,22 +754,35 @@ Two limitations are accepted:
two trees differ.
`trees` has four phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `read` counts the records, then reads
them in path order. It builds the directory tree and computes the digests as it
goes. `group` groups the directories by digest and picks the groups to report.
`write` writes them to stdout.
"Progress"). `open` opens the database. `read` gets the number of records from
the `totals` table, then reads them in path order. It builds the directory tree
and computes the digests as it goes. `group` groups the directories by digest
and picks the groups to report. `write` writes them to stdout.
#### Trees output format
The report is TSV on stdout, with a header line and then one row per duplicate
tree. A group of N trees gives N-1 rows. `files` is the number of regular files
in one copy of the tree, and `size` their total size in bytes:
tree. A group of N trees gives N-1 rows. After the `first` and `dupe` paths come
three numbers:
- `files`, the number of regular files in one copy of the tree;
- `size`, their total size in bytes;
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
added up.
```
first dupe files size
/srv/a/project /srv/backup/project 3417 104857600
first dupe files size disk_size
/srv/a/project /srv/backup/project 3417 104857600 61865984
```
A file's `disk_size` is the space its blocks take on disk. It can be more than
the file's size, because space is allocated in whole blocks. It is less than the
file's size on a compressed or sparse file. A sparse file has gaps that take no
space on disk. So two copies of one tree can differ in `disk_size`. When a file
in one copy is a hard link to one in the other, both copies count its blocks in
full. Removing it from one copy then frees nothing, as the limitation under
"`trees` mode" says.
The paths are escaped as described under "Report output format". The root
directory's path is `/`.
@@ -808,11 +829,11 @@ size of recorded files 150000000000000 (136.4 TiB)
of all recorded files, in bytes, then rounded to a unit such as TiB.
`info` has two phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `count` reads the schema version and
counts the records in one transaction. A transaction is a group of reads or
writes that SQLite carries out as one step, against the database as it stands at
one moment. So both describe the database at one moment, even while a scan is
writing. `info` writes its lines after the `count` display has ended.
"Progress"). `open` opens the database. `count` reads the schema version and the
row of `totals` (see "Schema") in one transaction. A transaction is a group of
reads or writes that SQLite carries out as one step, against the database as it
stands at one moment. So both describe the database at one moment, even while a
scan is writing. `info` writes its lines after the `count` display has ended.
### Database
@@ -908,33 +929,36 @@ Only one `scan` at a time can hold the lock.
- Then, before the walk as well, it writes:
- the `-wal` and `-shm` files, which stay beside the database while the scan
runs;
- in a new database, the table, the index and the schema version described
under "Schema".
- in a new database, the tables, the index, the triggers and the schema
version described under "Schema".
- During the `load` and `walk` phases, no records are written. A first scan's
database holds no records until the hash phase, however long its walk runs.
- During the `hash` phase, records are written 10,000 at a time.
- During the `update` phase, the rest of the hashed records are written, and the
deletions are made.
- During the `update` phase, the rest of the hashed records are written, and so
is the new `disk_size` of each unchanged file whose size on disk changed. Then
the deletions are made.
- As `scan` closes the database, the `-wal` and `-shm` files go away, unless a
report still has the database open.
#### Schema
The schema is the database's layout, its table and its index.
The schema is the database's layout: its tables, its index and its triggers. A
trigger is a statement SQLite runs by itself when a table changes.
`PRAGMA user_version` holds the schema version, currently 1. An empty database
is at version 0, and `scan` sets it up. Any other version is a fatal error.
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
error too. `scan` creates the table and the index and sets the version in one
transaction (see "`info` mode"). So the three land together or not at all. A
first scan stopped while doing so leaves an empty database, which the next scan
sets up. A database at version 0 that already has a `files` table was not made
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove
the file and rescan.
error too. `scan` creates the tables, the index and the triggers and sets the
version in one transaction (see "`info` mode"). So they all land together or not
at all. A first scan stopped while doing so leaves an empty database, which the
next scan sets up. A database at version 0 that already has a `files` table was
not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
remove the file and rescan.
```sql
CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512)
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
@@ -942,10 +966,28 @@ CREATE TABLE files (
content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
) WITHOUT ROWID;
CREATE INDEX files_signature ON files (size, head, tail, content);
CREATE TABLE totals (
records INTEGER NOT NULL, -- number of records in files
size INTEGER NOT NULL -- sizes of all records in files added up
);
INSERT INTO totals (records, size) VALUES (0, 0);
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
UPDATE totals SET records = records + 1, size = size + NEW.size;
END;
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
UPDATE totals SET records = records - 1, size = size - OLD.size;
END;
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
UPDATE totals SET size = size + NEW.size - OLD.size;
END;
```
- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
necessarily UTF-8 text.
- `disk_size` is the space the file's blocks take on disk. `lstat` is the system
call that stats a file without following a symlink. Its `st_blocks` field
counts that space in units of 512 bytes. `disk_size` is less than `size` on a
compressed or sparse file (see "Trees output format").
- `mtime` holds the whole seconds of the file's mtime, rounded down, and
`mtime_nsec` the nanoseconds past that second. Split this way, they hold any
mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
@@ -956,6 +998,16 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
not 32 bytes.
- `files_signature` lets `report` read the records in signature order without
sorting the table.
- `totals` holds one row: the number of records in `files` and the sum of their
sizes. Three triggers on `files` keep that row current. Each runs in the
transaction of the change to `files` that fires it, so the row and the records
always agree. `info` reads both numbers from this row. The `load` phase of
`scan` and the `read` phase of `report` and `trees` read the number of records
from it as their total. Counting the records in `files` would read every
record. That takes minutes on a database of tens of millions. The triggers
make each insert, each delete and each change of a record's `size` slower, so
a first scan of many small files takes about two fifths longer. On large
files, reading them takes most of the time, so the difference is small.
### Progress
@@ -966,8 +1018,9 @@ of its line are listed below.
The phases, and what each display counts:
- `scan` has four phases:
- `[1/4] load` counts the records first, then reads them against that count.
While it counts, the display says `counting the records`.
- `[1/4] load` gets the number of records from the `totals` table first,
then reads them against that number. Until it has the number, the display
says `counting the records`.
- `[2/4] walk` has no exact total. Its estimate is the number of records
under the operands, shown with a `~`. A scan with no records under its
operands, such as a first scan, has no estimate. The time remaining is
@@ -979,8 +1032,9 @@ The phases, and what each display counts:
- `report` has two phases:
- `[1/2] open` has no total. The display says `opening the database`, also
while it waits on a busy one.
- `[2/2] read` counts the records first, then reads them against that count.
It writes rows to stdout as it goes.
- `[2/2] read` gets the number of records from the `totals` table first,
then reads them against that number. Until it has the number, the display
says `counting the records`. It writes rows to stdout as it goes.
- `trees` has four phases:
- `[1/4] open` and `[2/4] read` are as for `report`, except that `read`
writes nothing.
@@ -988,9 +1042,8 @@ The phases, and what each display counts:
- `[4/4] write` has the duplicate-tree groups as its total.
- `info` has two phases:
- `[1/2] open` is as for `report`.
- `[2/2] count` has no total either, because one request to the database
counts everything. The display says `counting the records` until that
request returns.
- `[2/2] count` has no total either, because it reads one row. The display
says `counting the records` until it has the row.
Each display is one line holding:
@@ -1323,9 +1376,10 @@ All of the following, run in this directory, must pass:
- The groups come in order of size, largest first.
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
The row shows 2 files and 3100 bytes. The two `sub` directories are left
out, because the `t1` and `t2` group implies them. `t3` appears nowhere,
since one of its file names differs.
The row shows 2 files, 3100 bytes, and a `disk_size` that depends on the
filesystem. The two `sub` directories are left out, because the `t1` and
`t2` group implies them. `t3` appears nowhere, since one of its file names
differs.
After the rescan, the second `report` shows `one.bin` out of its group,
because its content changed. `copy.bin` and `copy2.bin` remain as one pair.