Add sfdupes info: show the database's details (closes #117)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
info prints the database's path and source, the sizes of the database file and its -wal file, when the database file last changed, the schema version and the record counts, one per line as a name, a tab and the value. It opens the database as report does and shows an open and a count phase on stderr. The schema version and the counts come from one read-only transaction, the counts from one query. seedDatabase now closes test databases out of WAL mode, as scan does; otherwise info's own connection creates an empty -wal file. The open phase's name and text became constants for the linter, and the scan command moved to newScanCommand to keep newRootCommand short enough. Judgement call: the lines are written after the last display ends, so they skip the display's stdout writer. Model: opus-5-5
This commit is contained in:
@@ -38,8 +38,9 @@ database persists between runs; a rescan only hashes files that are new or
|
||||
changed, or that may have gained a duplicate since the last scan, and removes
|
||||
records for files that no longer exist. `report` reads the database and prints
|
||||
the file-level duplicates report. `trees` reads the same database and prints the
|
||||
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
|
||||
with no `PATH` operand — prints a usage message and exits 2.
|
||||
duplicate-tree report. `info` prints the database's details: its path, the sizes
|
||||
of its files and the number of its records. A missing/invalid subcommand — or a
|
||||
`scan` invocation with no `PATH` operand — prints a usage message and exits 2.
|
||||
|
||||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
||||
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
||||
@@ -109,9 +110,9 @@ As a file `/etc/cron.d/sfdupes`:
|
||||
- A scan still running when the next one starts carries on. The new one fails at
|
||||
once, and the lines cron mails include
|
||||
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
|
||||
- `report` and `trees` need only read access to the database (see "Database").
|
||||
Under the usual umask of `022` the first scan creates it readable by every
|
||||
user, so an unprivileged user can run them against root's database.
|
||||
- `report`, `trees` and `info` need only read access to the database (see
|
||||
"Database"). Under the usual umask of `022` the first scan creates it readable
|
||||
by every user, so an unprivileged user can run them against root's database.
|
||||
|
||||
### Reading the reports
|
||||
|
||||
@@ -235,7 +236,7 @@ Goals, in order:
|
||||
|
||||
### Subcommands
|
||||
|
||||
Three subcommands, all implemented:
|
||||
Four subcommands, all implemented:
|
||||
|
||||
1. `scan` — walk the filesystem and synchronize the database: one signature
|
||||
record per regular file.
|
||||
@@ -243,11 +244,14 @@ Three subcommands, all implemented:
|
||||
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
|
||||
from the database records, compute a Merkle-style digest per directory, and
|
||||
report maximal groups of identical trees.
|
||||
4. `info` — the database's details: its path, the sizes of its files, its
|
||||
schema version and the counts of its records.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] PATH...
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
sfdupes --version
|
||||
sfdupes [command] --help
|
||||
```
|
||||
@@ -262,7 +266,7 @@ stdout.
|
||||
|
||||
### Database
|
||||
|
||||
All three subcommands operate on a single SQLite database file:
|
||||
All four subcommands operate on a single SQLite database file:
|
||||
|
||||
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
|
||||
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
|
||||
@@ -285,13 +289,13 @@ All three subcommands operate on a single SQLite database file:
|
||||
symlinks resolved. After a `cd` through a symlink to a directory, a path
|
||||
starting `../` therefore names a file beside the directory the symlink leads
|
||||
to, as it does for any other program. `scan` says `new database` when no
|
||||
file is at that path as it starts. `report` and `trees` print the line
|
||||
before they open the database, so the error for a missing database comes
|
||||
after it.
|
||||
file is at that path as it starts. `report`, `trees` and `info` print the
|
||||
line before they open the database, so the error for a missing database
|
||||
comes after it.
|
||||
|
||||
- `scan` creates the database (and its parent directory) on first use. `report`
|
||||
and `trees` require an existing database; a missing database file is a fatal
|
||||
error (exit 1) telling the user to run `scan` first.
|
||||
- `scan` creates the database (and its parent directory) on first use. `report`,
|
||||
`trees` and `info` require an existing database; a missing database file is a
|
||||
fatal error (exit 1) telling the user to run `scan` first.
|
||||
- Only one `scan` runs against a database at a time. For its whole run, `scan`
|
||||
holds an exclusive `flock(2)` lock on a lock file beside the database, named
|
||||
by appending `.lock` to the database's path with symlinks resolved
|
||||
@@ -307,8 +311,8 @@ All three subcommands operate on a single SQLite database file:
|
||||
on. The lock file is created on first use, open to its owner only, and left in
|
||||
place: a leftover file blocks nothing, because the lock ends with the process
|
||||
holding it however it ends, a fatal error or an interrupt included, and
|
||||
deleting the file while a scan runs would let a second scan start. `report`
|
||||
and `trees` never take the lock, so they run during a scan.
|
||||
deleting the file while a scan runs would let a second scan start. `report`,
|
||||
`trees` and `info` never take the lock, so they run during a scan.
|
||||
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so
|
||||
running a report while a cron `scan` is in progress is safe. The filesystem is
|
||||
authoritative; the database is an eventually-consistent reflection of it.
|
||||
@@ -341,10 +345,10 @@ All three subcommands operate on a single SQLite database file:
|
||||
- In the content phase: content hashes, again committed 10,000 at a time.
|
||||
- At the end, as it closes the database: the `-wal` and `-shm` files go
|
||||
away, unless a report still has the database open (see above).
|
||||
- `report` and `trees` open the database read-only and need only read access to
|
||||
the database file, and no write access to its directory. While the database is
|
||||
in WAL mode they also read the `-wal` and `-shm` files beside it, which SQLite
|
||||
creates with the database file's permissions.
|
||||
- `report`, `trees` and `info` open the database read-only and need only read
|
||||
access to the database file, and no write access to its directory. While the
|
||||
database is in WAL mode they also read the `-wal` and `-shm` files beside it,
|
||||
which SQLite creates with the database file's permissions.
|
||||
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
|
||||
with any other version is a fatal error. `scan` creates the schema and sets
|
||||
the version in one transaction, so a first scan stopped while doing so leaves
|
||||
@@ -727,13 +731,55 @@ Summary to stderr: records read, number of duplicate-tree groups, number of dupe
|
||||
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
|
||||
units.
|
||||
|
||||
### `info` mode
|
||||
|
||||
`info` takes no positional arguments and prints the database's details on
|
||||
stdout, one per line: a name, a tab, and the value. It opens the database the
|
||||
way `report` does, read-only and without the scan lock, so it runs during a scan
|
||||
and needs only read access (see "Database"). Like `report`, it never touches the
|
||||
scanned filesystem.
|
||||
|
||||
```
|
||||
database /var/lib/sfdupes/db.sqlite (default)
|
||||
database file size 2147483648 (2.0 GiB)
|
||||
-wal file size 4124152 (3.9 MiB)
|
||||
modified 2026-10-16T12:02:50Z
|
||||
schema version 1
|
||||
records 12345678
|
||||
records with head and tail hashes 1234567
|
||||
records with content hash 123456
|
||||
size of recorded files 150000000000000 (136.4 TiB)
|
||||
```
|
||||
|
||||
- `database` is the path and where it came from, as on the line every subcommand
|
||||
prints first on stderr (see "Database").
|
||||
- `database file size` and `-wal file size` are in bytes, then in human units.
|
||||
The `-wal` line is there only when a `-wal` file is beside the database, as
|
||||
while a scan runs.
|
||||
- `modified` is the time the database file was last changed, in UTC. While the
|
||||
database is in WAL mode, the newest changes may be in the `-wal` file alone.
|
||||
- `schema version` is the database's `PRAGMA user_version`. A database with
|
||||
another version than this build's is refused, as by `report`.
|
||||
- `records` is the number of records. `records with head and tail hashes` and
|
||||
`records with content hash` count the records that have those hashes (see
|
||||
"Database"), and `size of recorded files` adds up the sizes of all records, in
|
||||
bytes, then in human units.
|
||||
|
||||
`info` runs in two phases, each with its own progress display on stderr (see
|
||||
"Progress"): `open` opens the database; `count` reads the schema version and
|
||||
counts the records with one query, a single pass over the table, both in one
|
||||
read-only transaction, so the numbers agree with each other while a scan is
|
||||
committing. `info` writes its lines once the `count` display has ended, so on a
|
||||
terminal they come after the displays.
|
||||
|
||||
### Progress
|
||||
|
||||
Use the progress-bar library for all progress of `scan`, `report` and `trees`;
|
||||
rendering in the style of `pv` is the model. All progress goes to stderr.
|
||||
Use the progress-bar library for all progress of `scan`, `report`, `trees` and
|
||||
`info`; rendering in the style of `pv` is the model. All progress goes to
|
||||
stderr.
|
||||
|
||||
`scan`, `report` and `trees` must always show that they are working and not
|
||||
hung. Each phase gets its own display from the moment it starts, naming the
|
||||
`scan`, `report`, `trees` and `info` must always show that they are working and
|
||||
not hung. Each phase gets its own display from the moment it starts, naming the
|
||||
phase and its place among the command's phases and showing its progress, its
|
||||
elapsed time and its estimated time remaining (ETA).
|
||||
|
||||
@@ -773,6 +819,14 @@ A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
|
||||
the rows to stdout as it goes. While the count runs, the display says it is
|
||||
counting the records.
|
||||
|
||||
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
|
||||
|
||||
- `open` has no total. While it opens the database, waiting on a busy one
|
||||
included, the display says it is opening the database.
|
||||
- `count` has no total either, because one query counts everything: the display
|
||||
says it is counting the records until the query returns, and its ETA is
|
||||
unknown.
|
||||
|
||||
Each display is one line holding:
|
||||
|
||||
- the phase and its place among the phases
|
||||
@@ -792,13 +846,14 @@ Example shape (exact layout is flexible, content is not):
|
||||
|
||||
Additional requirements:
|
||||
|
||||
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`
|
||||
and the `open` phase of `report` and `trees`, once it has counted what it will
|
||||
do) is drawn as its line followed by the progress-bar library's bar, 10
|
||||
columns wide, which shows the percent in place of the line's own. Any other
|
||||
phase, and a phase whose bar has no room, is drawn as the library's spinner
|
||||
followed by its line. Each display is drawn again several times a second, so
|
||||
its elapsed time keeps moving while a phase waits for its next item.
|
||||
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
|
||||
the `open` phase of `report` and `trees`, and both phases of `info`, once it
|
||||
has counted what it will do) is drawn as its line followed by the progress-bar
|
||||
library's bar, 10 columns wide, which shows the percent in place of the line's
|
||||
own. Any other phase, and a phase whose bar has no room, is drawn as the
|
||||
library's spinner followed by its line. Each display is drawn again several
|
||||
times a second, so its elapsed time keeps moving while a phase waits for its
|
||||
next item.
|
||||
- On a terminal no line reaches the terminal's last column, however narrow the
|
||||
terminal is. The phase and its place, the progress, the elapsed time and the
|
||||
ETA are kept whole: a line too wide for the terminal leaves out its rate first
|
||||
@@ -831,10 +886,12 @@ Additional requirements:
|
||||
- `0`: success, even if individual files were skipped with warnings.
|
||||
- `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
|
||||
already running against the same database, the database cannot be
|
||||
created/opened/read/written, a missing database for `report`/`trees`, stdout
|
||||
write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see below).
|
||||
created/opened/read/written, a missing database for `report`/`trees`/`info`,
|
||||
stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
|
||||
below).
|
||||
- `2`: usage error (including `scan` with no `PATH` operand, `scan` with
|
||||
`--workers` below 1, and `report`/`trees` with any positional argument).
|
||||
`--workers` below 1, and `report`/`trees`/`info` with any positional
|
||||
argument).
|
||||
|
||||
A stdout write failure, such as a full disk, is reported in one line on stderr
|
||||
and exits 1. Two cases never reach sfdupes as a failed write:
|
||||
|
||||
Reference in New Issue
Block a user