Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
468f0cb844 |
@@ -106,9 +106,10 @@ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
||||
|
||||
### Daily scan from `cron`
|
||||
|
||||
Run `scan` as root, so that it can read every file. A path that it cannot read
|
||||
is skipped with a warning and loses its database record. As the file
|
||||
`/etc/cron.d/sfdupes`:
|
||||
Run `scan` as root, so that it can read every file. `scan` warns about a
|
||||
directory it cannot read, and about a new or changed file it cannot read. It
|
||||
skips the path and deletes the records at or below it (see "Rules for the
|
||||
walk"). As the file `/etc/cron.d/sfdupes`:
|
||||
|
||||
```
|
||||
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
||||
@@ -348,10 +349,7 @@ default. An operand may be a directory or a regular file.
|
||||
- Each operand is made absolute before the walk. A relative operand is joined to
|
||||
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
|
||||
`..` in it is resolved as text, and symlinks are not followed. Each record is
|
||||
stored under that absolute path. After a `cd` through a symlink, `scan .`
|
||||
names the symlink and is skipped as a symlink operand. `scan sub` in that
|
||||
directory records its files under the symlink's path. `report` can then list
|
||||
each as a duplicate of the same file recorded under its real path.
|
||||
stored under that absolute path.
|
||||
- All operands belong to one scan, and one walk covers them all at once (see
|
||||
"Concurrency").
|
||||
- An operand that repeats another is dropped before the walk, without a line.
|
||||
@@ -376,7 +374,10 @@ default. An operand may be a directory or a regular file.
|
||||
`scan` prints a warning naming the path and what it is. It counts the operand
|
||||
as skipped and drops it. An operand beneath a symlink operand is scanned all
|
||||
the same. The scan keeps the records beneath a dropped operand, unless another
|
||||
operand covers them.
|
||||
operand covers them. After a `cd` through a symlink, `scan .` names the
|
||||
symlink and is skipped as a symlink operand. `scan sub` in that directory
|
||||
records its files under the symlink's path. `report` can then list each as a
|
||||
duplicate of the same file recorded under its real path.
|
||||
- An operand at or inside a directory named `.zfs` is never scanned, whether
|
||||
given alone or under another operand. The check looks at the path as given and
|
||||
at the path with its symlinks resolved. `scan` prints a warning naming the
|
||||
@@ -404,8 +405,9 @@ operands:
|
||||
- The scan deletes each record under a scanned operand whose file it could not
|
||||
verify. That covers deleted files, and paths the scan could not stat or read
|
||||
this run. To stat a file is to read its details, such as its size, without
|
||||
opening it. So the database holds only what the latest complete scan over each
|
||||
path could verify. A later successful scan adds such files again.
|
||||
opening it. So after a complete scan, the database holds only what that scan
|
||||
could verify under its operands. A later successful scan adds such files
|
||||
again.
|
||||
- Records outside the scanned operands are left alone, so separate trees can be
|
||||
scanned on different schedules into one database.
|
||||
- The scan keeps every record at or below a path that the walk leaves out for
|
||||
@@ -514,11 +516,10 @@ walk practical on large or busy storage pools.
|
||||
The path is not counted as skipped. The records below it are kept, as
|
||||
described under "What a scan does to the database".
|
||||
|
||||
- Below an operand, the walk does not enter a default exclude. The default
|
||||
excludes hold temporary files and the kernel's view of the running system, so
|
||||
that `sfdupes scan /` walks only the files worth comparing. They depend on the
|
||||
system, and `sfdupes scan --help` lists those of the system it runs on. On
|
||||
Linux they are:
|
||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||
default excludes hold temporary files and the kernel's view of the running
|
||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
||||
the system it runs on. On Linux they are:
|
||||
|
||||
```
|
||||
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
||||
@@ -547,17 +548,22 @@ walk practical on large or busy storage pools.
|
||||
scan: skipped /var/tmp (default exclude)
|
||||
```
|
||||
|
||||
The directory is not counted as skipped. A default exclude on a filesystem
|
||||
that is not persistent, such as `/proc`, gets the line for its filesystem
|
||||
instead. One that is itself an operand gets no line, because it is walked as
|
||||
that operand. The scan keeps the records below a default exclude that the
|
||||
The directory is not counted as skipped. `scan` prints the filesystem line
|
||||
instead for a default exclude that is not on a persistent filesystem, such
|
||||
as `/proc`. One that is itself an operand gets no line, because it is walked
|
||||
as that operand. The scan keeps the records below a default exclude that the
|
||||
walk did not enter, as described under "What a scan does to the database".
|
||||
|
||||
- A path that cannot be read, for lack of permission or any other error, gets a
|
||||
one-line warning on stderr and is skipped. The scan continues, and the summary
|
||||
line says how many paths were skipped. A skipped path that has a record from
|
||||
an earlier scan loses that record, and so does every path under an unreadable
|
||||
directory. The database mirrors what the latest complete scan could verify.
|
||||
- `scan` skips each of these, with a one-line warning on stderr:
|
||||
- a directory it cannot read;
|
||||
- a path it cannot stat;
|
||||
- a new or changed file it cannot read.
|
||||
|
||||
An unchanged file is not opened, so it keeps its record even when it has
|
||||
become unreadable. The scan continues, and the summary line says how many
|
||||
paths were skipped. A skipped path that has a record from an earlier scan
|
||||
loses that record, and so does every path under an unreadable directory.
|
||||
|
||||
- A file or directory that disappears while the scan runs is not an error. The
|
||||
scan prints no line for it and does not count it as skipped. A path gone when
|
||||
the walk reaches it gets no record, and so does a file gone when the hash
|
||||
@@ -580,8 +586,8 @@ from a thread of its own and never blocks the workers.
|
||||
|
||||
- one line naming the database (see "Database");
|
||||
- for each dropped operand, in operand order, the warning or the line described
|
||||
under "Operands". An operand dropped as a repeat of another, or as lying under
|
||||
another, gets neither;
|
||||
under "Operands". `scan` prints neither for an operand that repeats another or
|
||||
lies under another;
|
||||
- the progress display of each phase (see "Progress"). `scan` prints each
|
||||
warning for a skipped path as it happens. It prints a line the same way for
|
||||
each path left out as a default exclude or as not on a persistent filesystem;
|
||||
@@ -715,9 +721,9 @@ Processing:
|
||||
directories is a duplicate-tree group.
|
||||
- Only the outermost duplicate trees are reported. A group is left out when its
|
||||
members' parents are distinct directories that all have the same digest. The
|
||||
parents' group, or an ancestor's, already implies it. A group holding sibling
|
||||
directories, or members whose parents have different digests, is always
|
||||
reported.
|
||||
parents' group, or an ancestor's, already implies it. A group is always
|
||||
reported when it holds sibling directories, or when its members' parents have
|
||||
different digests.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by total tree size, largest first, then by `first` path.
|
||||
@@ -1063,8 +1069,8 @@ Each phase prints its line:
|
||||
- every 5 seconds while it runs, whether or not an item has completed;
|
||||
- once more as it ends.
|
||||
|
||||
A phase that stops short of its total, as an interrupted one does, keeps showing
|
||||
the count it stopped at.
|
||||
When a phase stops short of its total, as an interrupted one does, it keeps
|
||||
showing the count it stopped at.
|
||||
|
||||
### Error handling and exit codes
|
||||
|
||||
@@ -1128,8 +1134,8 @@ the repository root itself, so it can be run from any directory.
|
||||
- An installed `node` is used when its major version is 22. A `yarn` already
|
||||
on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
|
||||
activated through `corepack`, the `yarn` installer that ships with `node`.
|
||||
When there is no `corepack` either, `npm`, the package manager that ships
|
||||
with `node`, installs it.
|
||||
When there is no `corepack` either, the script installs `yarn` with `npm`,
|
||||
the package manager that ships with `node`.
|
||||
- Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
|
||||
version manager, and activates `yarn` 1.22.22 under that `node` through
|
||||
`corepack`. When `nvm` is missing, the script first downloads it and
|
||||
@@ -1160,8 +1166,8 @@ the repository root itself, so it can be run from any directory.
|
||||
for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
|
||||
writes no image. The stage starts from the `golangci/golangci-lint` image,
|
||||
pinned by its digest, and runs three build steps:
|
||||
- the `gofmt` check, which fails on a Go source that `gofmt`, Go's own
|
||||
formatter, would change;
|
||||
- the `gofmt` check, which fails when `gofmt`, Go's own formatter, would
|
||||
change a Go source;
|
||||
- `golangci-lint config verify`, which checks the linter's settings file
|
||||
`.golangci.yml`;
|
||||
- `golangci-lint run`, which lints the Go sources.
|
||||
|
||||
Reference in new issue
Block a user