From f373b7d9e4973bdb2f0cc4945bf20ccae0af6a07 Mon Sep 17 00:00:00 2001 From: sneak Date: Mon, 2 Nov 2026 21:35:33 +0000 Subject: [PATCH] Carry the README changes from next into the rewrite `next` changed the old README after this branch was cut, in https://git.eeqj.de/sneak/sfdupes/issues/146 (the `totals` table, commit 98193b2) and https://git.eeqj.de/sneak/sfdupes/issues/147 (`disk_size`, commit 4ed2e37). This commit states the same facts in the rewritten README, in the places https://git.eeqj.de/sneak/sfdupes/pulls/140#issuecomment-163070 lists, and changes nothing else. Every statement was checked against the code on `next`. Model: fable-5-1 --- README.md | 148 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 101 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index a8821f5..c1b0fe3 100644 --- a/README.md +++ b/README.md @@ -405,6 +405,11 @@ operands: - A file already in the database is not read again when its size and mtime match its record. The mtime matches when it is not newer than the recorded one. This is what makes a daily rescan cheap. +- When such a file's size on disk differs from its recorded `disk_size`, `scan` + updates `disk_size` alone. It still does not read the file, and counts it as + unchanged. ZFS, for example, counts a new file's blocks only seconds after the + write. A block is the unit in which a filesystem allocates disk space. A file + scanned within those seconds gets its real `disk_size` at the next scan. - `scan` hashes a file again when its size differs or its mtime is newer. The mtime is compared to the nanosecond, as finely as the filesystem records it. A same-size rewrite counts as a change whenever the filesystem gives it a later @@ -435,15 +440,17 @@ operands: phases. 1. `load` reads the existing records. Each record under the operands is kept in - memory by path, with its size and mtime, for the walk to compare against. + memory by path, with its size, its size on disk and its mtime, for the walk + to compare against. 2. `walk` lists the trees under all operands. Several workers share one queue of directories to read. A worker is one thread of the program. A thread is one of several parts of a program that run at the same time and share its memory. "Concurrency" says how many workers there are. Each worker reads one directory at a time. It hands the subdirectories it finds back to the queue, and it stats each regular file as it finds it. The walk compares each file - with its record. An unchanged file needs nothing more. Every new or changed - file is passed to the hash phase. + with its record. An unchanged file is not read again. When its size on disk + has changed, the update phase writes its new `disk_size`. Every new or + changed file is passed to the hash phase. 3. `hash` reads every new or changed file and computes its three hashes. Files are read in inode order. An inode is the filesystem's own record of a file. Files with nearby inode numbers usually lie near each other on disk. Reading @@ -452,8 +459,9 @@ phases. tree made of hard links therefore costs one read per file. Completed records are written to the database 10,000 at a time while hashing runs. So a scan interrupted after hours keeps everything hashed so far. -4. `update` writes the hashed records not yet written, then deletes the records - the scan did not verify. +4. `update` writes the hashed records not yet written, and the new `disk_size` + of each unchanged file whose size on disk changed. Then it deletes the + records the scan did not verify. Reading a tree one directory at a time waits on the disk at every step and takes hours at millions of files. Reading many directories at once is what makes the @@ -656,14 +664,14 @@ Processing: the same for the same database contents. `report` has two phases, each with its own progress display on stderr (see -"Progress"). `open` opens the database. `read` counts the records, then reads -them in signature order, largest size first. The database keeps an index named -`files_signature`, a sorted list of the records by signature (see "Schema"). -Reading it backwards gives that order, so SQLite sorts nothing and returns the -first record at once. Records with one signature come together, and so do the -records of one size. So `report` holds only the duplicate groups of the size it -is reading. Once it reaches a smaller size, it writes them, ordered by `first` -path. +"Progress"). `open` opens the database. `read` gets the number of records from +the `totals` table (see "Schema"), then reads them in signature order, largest +size first. The database keeps an index named `files_signature`, a sorted list +of the records by signature (see "Schema"). Reading it backwards gives that +order, so SQLite sorts nothing and returns the first record at once. Records +with one signature come together, and so do the records of one size. So `report` +holds only the duplicate groups of the size it is reading. Once it reaches a +smaller size, it writes them, ordered by `first` path. #### Report output format @@ -746,22 +754,35 @@ Two limitations are accepted: two trees differ. `trees` has four phases, each with its own progress display on stderr (see -"Progress"). `open` opens the database. `read` counts the records, then reads -them in path order. It builds the directory tree and computes the digests as it -goes. `group` groups the directories by digest and picks the groups to report. -`write` writes them to stdout. +"Progress"). `open` opens the database. `read` gets the number of records from +the `totals` table, then reads them in path order. It builds the directory tree +and computes the digests as it goes. `group` groups the directories by digest +and picks the groups to report. `write` writes them to stdout. #### Trees output format The report is TSV on stdout, with a header line and then one row per duplicate -tree. A group of N trees gives N-1 rows. `files` is the number of regular files -in one copy of the tree, and `size` their total size in bytes: +tree. A group of N trees gives N-1 rows. After the `first` and `dupe` paths come +three numbers: + +- `files`, the number of regular files in one copy of the tree; +- `size`, their total size in bytes; +- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size` + added up. ``` -first dupe files size -/srv/a/project /srv/backup/project 3417 104857600 +first dupe files size disk_size +/srv/a/project /srv/backup/project 3417 104857600 61865984 ``` +A file's `disk_size` is the space its blocks take on disk. It can be more than +the file's size, because space is allocated in whole blocks. It is less than the +file's size on a compressed or sparse file. A sparse file has gaps that take no +space on disk. So two copies of one tree can differ in `disk_size`. When a file +in one copy is a hard link to one in the other, both copies count its blocks in +full. Removing it from one copy then frees nothing, as the limitation under +"`trees` mode" says. + The paths are escaped as described under "Report output format". The root directory's path is `/`. @@ -808,11 +829,11 @@ size of recorded files 150000000000000 (136.4 TiB) of all recorded files, in bytes, then rounded to a unit such as TiB. `info` has two phases, each with its own progress display on stderr (see -"Progress"). `open` opens the database. `count` reads the schema version and -counts the records in one transaction. A transaction is a group of reads or -writes that SQLite carries out as one step, against the database as it stands at -one moment. So both describe the database at one moment, even while a scan is -writing. `info` writes its lines after the `count` display has ended. +"Progress"). `open` opens the database. `count` reads the schema version and the +row of `totals` (see "Schema") in one transaction. A transaction is a group of +reads or writes that SQLite carries out as one step, against the database as it +stands at one moment. So both describe the database at one moment, even while a +scan is writing. `info` writes its lines after the `count` display has ended. ### Database @@ -908,33 +929,36 @@ Only one `scan` at a time can hold the lock. - Then, before the walk as well, it writes: - the `-wal` and `-shm` files, which stay beside the database while the scan runs; - - in a new database, the table, the index and the schema version described - under "Schema". + - in a new database, the tables, the index, the triggers and the schema + version described under "Schema". - During the `load` and `walk` phases, no records are written. A first scan's database holds no records until the hash phase, however long its walk runs. - During the `hash` phase, records are written 10,000 at a time. -- During the `update` phase, the rest of the hashed records are written, and the - deletions are made. +- During the `update` phase, the rest of the hashed records are written, and so + is the new `disk_size` of each unchanged file whose size on disk changed. Then + the deletions are made. - As `scan` closes the database, the `-wal` and `-shm` files go away, unless a report still has the database open. #### Schema -The schema is the database's layout, its table and its index. +The schema is the database's layout: its tables, its index and its triggers. A +trigger is a statement SQLite runs by itself when a table changes. `PRAGMA user_version` holds the schema version, currently 1. An empty database is at version 0, and `scan` sets it up. Any other version is a fatal error. `report`, `trees` and `info` set nothing up, so for them version 0 is a fatal -error too. `scan` creates the table and the index and sets the version in one -transaction (see "`info` mode"). So the three land together or not at all. A -first scan stopped while doing so leaves an empty database, which the next scan -sets up. A database at version 0 that already has a `files` table was not made -by `sfdupes`. Every subcommand refuses it, with an error telling you to remove -the file and rescan. +error too. `scan` creates the tables, the index and the triggers and sets the +version in one transaction (see "`info` mode"). So they all land together or not +at all. A first scan stopped while doing so leaves an empty database, which the +next scan sets up. A database at version 0 that already has a `files` table was +not made by `sfdupes`. Every subcommand refuses it, with an error telling you to +remove the file and rescan. ```sql CREATE TABLE files ( path BLOB PRIMARY KEY, -- absolute path, raw bytes size INTEGER NOT NULL, -- bytes + disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512) mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB @@ -942,10 +966,28 @@ CREATE TABLE files ( content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples ) WITHOUT ROWID; CREATE INDEX files_signature ON files (size, head, tail, content); +CREATE TABLE totals ( + records INTEGER NOT NULL, -- number of records in files + size INTEGER NOT NULL -- sizes of all records in files added up +); +INSERT INTO totals (records, size) VALUES (0, 0); +CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN + UPDATE totals SET records = records + 1, size = size + NEW.size; +END; +CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN + UPDATE totals SET records = records - 1, size = size - OLD.size; +END; +CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN + UPDATE totals SET size = size + NEW.size - OLD.size; +END; ``` - `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not necessarily UTF-8 text. +- `disk_size` is the space the file's blocks take on disk. `lstat` is the system + call that stats a file without following a symlink. Its `st_blocks` field + counts that space in units of 512 bytes. `disk_size` is less than `size` on a + compressed or sparse file (see "Trees output format"). - `mtime` holds the whole seconds of the file's mtime, rounded down, and `mtime_nsec` the nanoseconds past that second. Split this way, they hold any mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit @@ -956,6 +998,16 @@ CREATE INDEX files_signature ON files (size, head, tail, content); not 32 bytes. - `files_signature` lets `report` read the records in signature order without sorting the table. +- `totals` holds one row: the number of records in `files` and the sum of their + sizes. Three triggers on `files` keep that row current. Each runs in the + transaction of the change to `files` that fires it, so the row and the records + always agree. `info` reads both numbers from this row. The `load` phase of + `scan` and the `read` phase of `report` and `trees` read the number of records + from it as their total. Counting the records in `files` would read every + record. That takes minutes on a database of tens of millions. The triggers + make each insert, each delete and each change of a record's `size` slower, so + a first scan of many small files takes about two fifths longer. On large + files, reading them takes most of the time, so the difference is small. ### Progress @@ -966,8 +1018,9 @@ of its line are listed below. The phases, and what each display counts: - `scan` has four phases: - - `[1/4] load` counts the records first, then reads them against that count. - While it counts, the display says `counting the records`. + - `[1/4] load` gets the number of records from the `totals` table first, + then reads them against that number. Until it has the number, the display + says `counting the records`. - `[2/4] walk` has no exact total. Its estimate is the number of records under the operands, shown with a `~`. A scan with no records under its operands, such as a first scan, has no estimate. The time remaining is @@ -979,8 +1032,9 @@ The phases, and what each display counts: - `report` has two phases: - `[1/2] open` has no total. The display says `opening the database`, also while it waits on a busy one. - - `[2/2] read` counts the records first, then reads them against that count. - It writes rows to stdout as it goes. + - `[2/2] read` gets the number of records from the `totals` table first, + then reads them against that number. Until it has the number, the display + says `counting the records`. It writes rows to stdout as it goes. - `trees` has four phases: - `[1/4] open` and `[2/4] read` are as for `report`, except that `read` writes nothing. @@ -988,9 +1042,8 @@ The phases, and what each display counts: - `[4/4] write` has the duplicate-tree groups as its total. - `info` has two phases: - `[1/2] open` is as for `report`. - - `[2/2] count` has no total either, because one request to the database - counts everything. The display says `counting the records` until that - request returns. + - `[2/2] count` has no total either, because it reads one row. The display + says `counting the records` until it has the row. Each display is one line holding: @@ -1323,9 +1376,10 @@ All of the following, run in this directory, must pass: - The groups come in order of size, largest first. `trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`. - The row shows 2 files and 3100 bytes. The two `sub` directories are left - out, because the `t1` and `t2` group implies them. `t3` appears nowhere, - since one of its file names differs. + The row shows 2 files, 3100 bytes, and a `disk_size` that depends on the + filesystem. The two `sub` directories are left out, because the `t1` and + `t2` group implies them. `t3` appears nowhere, since one of its file names + differs. After the rescan, the second `report` shows `one.bin` out of its group, because its content changed. `copy.bin` and `copy2.bin` remain as one pair.