1 Commits
Author SHA1 Message Date
sneak 18aa3a383d scan: skip OS default excludes, and record nothing on filesystems that are not persistent (closes #124)
check / check (push) Canceled after 0s
scan never records anything on a filesystem that is not persistent
(tmpfs, proc, sysfs and the other kernel or memory-backed types). As it
starts it notes the device ID of each such mount point, from
/proc/self/mountinfo on Linux and the mounted-filesystem list on macOS,
and checks every operand, directory and file the walk meets against
them; no flag turns this off. Below an operand the walk also skips the
default excludes, a path list per system (Linux, macOS, and Windows,
which is not used yet), and --no-default-excludes turns off only those.
A default exclude that is itself an operand gets no skip line. The tests
stop at once, saying why, when TMPDIR is not on a persistent filesystem.

Judgement call: an unreadable mount list fails the scan, and no flag gets round it.
Judgement call: a file on such a filesystem below an operand gets the same line as a directory.
Judgement call: an operand on such a filesystem that another operand's walk also reaches is named twice.
Unverified: the macOS mount list is built for macOS but never run; bind mounts need root to test.

Model: opus-5-5
2026-10-17 00:31:52 +00:00
19 changed files with 241 additions and 787 deletions
+11 -19
View File
@@ -1,23 +1,16 @@
# In every stage ARG CACHEBUST comes right after the steps that install # Every stage declares ARG CACHEBUST first: script/ passes a new value
# system packages and download Go's modules, or right after FROM in a # on every build, which reruns every RUN step after it. The steps that
# stage with neither, and every step that lints, tests or compiles # compile mount Go's build cache, which survives from build to build.
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build. The
# test phase runs as nobody and so mounts Go's build cache at
# /tmp/.cache/go-build; its own comment says why.
# Lint phase, built alone by script/lint. The tools are invoked directly # Lint phase, built alone by script/lint. The tools are invoked directly
# rather than through `make lint`, which runs docker itself and so cannot # rather than through `make lint`, which runs docker itself and so cannot
# run inside a build step. # run inside a build step.
# golangci/golangci-lint:v2.14.0, 2026-10-07 # golangci/golangci-lint:v2.14.0, 2026-10-07
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
ARG CACHEBUST
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
# The gofmt half of `make fmt-check`. gofmt's output is assigned to a # The gofmt half of `make fmt-check`. gofmt's output is assigned to a
@@ -31,7 +24,6 @@ RUN files="$(gofmt -s -l .)" && \
# Validates .golangci.yml against the schema the pinned binary embeds. # Validates .golangci.yml against the schema the pinned binary embeds.
RUN golangci-lint config verify --config .golangci.yml RUN golangci-lint config verify --config .golangci.yml
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./... golangci-lint run --config .golangci.yml ./...
# Test phase, built alone by script/test. -race needs cgo and so a C # Test phase, built alone by script/test. -race needs cgo and so a C
@@ -46,12 +38,12 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
# nobody (65534) so the tests can write to it. # nobody (65534) so the tests can write to it.
# golang:1.25-trixie, 2026-10-04 # golang:1.25-trixie, 2026-10-04
FROM golang@sha256:2c4c60ef415fbfa5e90300722293bef36c5e63fae17570ce18f580af933dbd73 AS test FROM golang@sha256:2c4c60ef415fbfa5e90300722293bef36c5e63fae17570ce18f580af933dbd73 AS test
ARG CACHEBUST
USER nobody USER nobody
ENV HOME=/tmp ENV HOME=/tmp
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN --mount=type=cache,target=/tmp/.cache/go-build,uid=65534,gid=65534 \ RUN --mount=type=cache,target=/tmp/.cache/go-build,uid=65534,gid=65534 \
go test -count=1 -timeout 90s -race -cover ./... || \ go test -count=1 -timeout 90s -race -cover ./... || \
@@ -59,19 +51,19 @@ RUN --mount=type=cache,target=/tmp/.cache/go-build,uid=65534,gid=65534 \
go test -count=1 -timeout 90s -race -v ./...; exit 1; } go test -count=1 -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies are # Build stage. Nothing is wanted from either phase above; the copies are
# what make BuildKit build them first, so nothing after them can run # what make BuildKit build them first, so this stage cannot run unless
# unless lint and test passed. # lint and test passed.
# golang:1.25-alpine, 2026-07-23 # golang:1.25-alpine, 2026-07-23
FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
RUN apk add --no-cache git make
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git make
# A tar-stream context keeps the sender's file owners, which git refuses. # A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . . COPY . .
# The version stamped into the binary: the VERSION build argument when # The version stamped into the binary: the VERSION build argument when
+43 -87
View File
@@ -49,40 +49,15 @@ results are as fresh as the last completed scan.
### Install ### Install
With Go installed, this builds and installs the newest release: With Go installed, this builds and installs the current `main` branch:
```sh
go install sneak.berlin/go/sfdupes@latest
```
Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
`@main` installs the tip before it is tagged:
```sh ```sh
go install sneak.berlin/go/sfdupes@main go install sneak.berlin/go/sfdupes@main
``` ```
`v0.0.1` is an early build without a database. It does not work as this README The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. A
describes, and `go.mod` retracts it. binary installed this way reports its version as `dev`; one built from a clone
or into the Docker image carries the git tag or commit it was built from.
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
see which version an installed binary is:
```sh
sfdupes --version
go version -m "$(command -v sfdupes)"
```
`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
or for `@main` a pseudo-version ending in the first 12 characters of the
commit's hash. A binary built from a clone or into the Docker image carries the
git tag or commit it was built from instead, and `VERSION` is `dev` only when
the binary carries no version at all. `go version -m` prints what Go recorded in
the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
and earlier print `dev`, so check those with `go version -m`.
From a clone, `make build` writes the binary to `./sfdupes`: From a clone, `make build` writes the binary to `./sfdupes`:
@@ -287,10 +262,10 @@ sfdupes [command] --help
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem, and `--no-default-excludes` lets the walk enter the operand's filesystem, and `--no-default-excludes` lets the walk enter the
default excludes, a list of paths; all three are described under "`scan` mode". default excludes, a list of paths; all three are described under "`scan` mode".
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see `sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`, exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
alone or after a subcommand, prints the help text to stderr and exits 0, writing subcommand, prints the help text to stderr and exits 0, writing nothing to
nothing to stdout. stdout.
### Database ### Database
@@ -521,9 +496,8 @@ operands:
can be scanned on different schedules into the same database. The one can be scanned on different schedules into the same database. The one
exception is the content phase below: a stored file of 10 MiB or more without exception is the content phase below: a stored file of 10 MiB or more without
a `content` hash is read for one, wherever it lies, once its size, `head`, and a `content` hash is read for one, wherever it lies, once its size, `head`, and
`tail` match another record's. If that file is gone, has changed since its `tail` match another record's. If that file is gone or has changed since its
record was written, or is not on a persistent filesystem, it is not read and record was written, the record is left as it is.
the record is left as it is.
- A record at or below a path the walk left out, because that path is a default - A record at or below a path the walk left out, because that path is a default
exclude, is not on a persistent filesystem, or, with `-x`, is a directory on exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
another filesystem than its operand, counts as outside the scanned operands: another filesystem than its operand, counts as outside the scanned operands:
@@ -576,20 +550,19 @@ the hash phase:
be read are kept in memory, never every file's hashes. Every record sharing be read are kept in memory, never every file's hashes. Every record sharing
their size, `head`, and `tail`, including one that already has a `content` their size, `head`, and `tail`, including one that already has a `content`
hash, has its file checked with `lstat` first. A file that is gone, is no hash, has its file checked with `lstat` first. A file that is gone, is no
longer a regular file, has changed (a different size, or an mtime newer than longer a regular file, or has changed (a different size, or an mtime newer
recorded), or is not on a persistent filesystem (see "Rules for the walk") than recorded) keeps its record as it is and does not count as a match for
keeps its record as it is and does not count as a match for the others. Any the others. Any other `lstat` error is warned about and counted as skipped,
other `lstat` error is warned about and counted as skipped, with the same with the same result. If such a record has no `content` hash, it stays out
result. If such a record has no `content` hash, it stays out of duplicate of duplicate groups; if it has one, it is still reported until a scan
groups; if it has one, it is still reported until a scan covering its own covering its own tree updates or removes it. The files that pass and have no
tree updates or removes it. The files that pass and have no `content` hash `content` hash are read only if at least two of those records pass, so a
are read only if at least two of those records pass, so a file whose only file whose only matches are stale costs no read; a file that already has a
matches are stale costs no read; a file that already has a `content` hash is `content` hash is never read again. They are read by a worker pool as in the
never read again. They are read by a worker pool as in the hash phase, in hash phase, in inode order and once per inode, and their content hashes are
inode order and once per inode, and their content hashes are committed in committed in batches. A failed read is warned about and counted as skipped;
batches. A failed read is warned about and counted as skipped; its record its record keeps an empty `content`, so it is not a duplicate, and a later
keeps an empty `content`, so it is not a duplicate, and a later scan tries scan tries again.
again.
Rules for the walk: Rules for the walk:
@@ -600,15 +573,6 @@ Rules for the walk:
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking - Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
them would list every file once per snapshot), not even when it is an operand; them would list every file once per snapshot), not even when it is an operand;
such an operand is dropped the same way. such an operand is dropped the same way.
- Never record the scan's own database. Once the database and its `.lock` file
are open, `scan` notes the device and inode of the database file, its `-wal`
and `-shm` files and its `.lock` file, and the walk passes over a file with
one of them, however it reaches it: as an operand, below one, or through a
symlinked directory. It prints no line for such a file and does not count it
as skipped. A record that an earlier scan left for one of them is deleted,
like the record of any file the scan no longer verifies. There is no
`-journal` file during the walk, because the database is in WAL mode then (see
"Database").
- Filesystem boundaries are crossed by default. With `-x` (long form - Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend `--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each into a directory on a different filesystem than its `PATH` operand; each
@@ -625,18 +589,15 @@ Rules for the walk:
it names the database and before it takes the lock, `scan` lists the mounted it names the database and before it takes the lock, `scan` lists the mounted
filesystems, from `/proc/self/mountinfo` on Linux and from the system's list filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
of mounted filesystems on macOS (on any other system it lists none), and notes of mounted filesystems on macOS (on any other system it lists none), and notes
the device ID of each one of those types. On Linux that is the device ID the the device ID of the mount point of each one of those types, read with
list gives, so a filesystem that a later mount hides still gives its own. On `lstat`. Where one filesystem is mounted over another, the one mounted last
macOS it is read with `lstat` on the mount point, and where one filesystem is counts. If `scan` cannot list them, it fails (exit 1). The walk then never
mounted over another on the same mount point, the one mounted last counts. If enters a directory, and never records a file, on one of those devices, however
`scan` cannot list them, it fails (exit 1). The walk then never enters a it reaches it: as an operand, below one, or through a bind mount, which keeps
directory, and never records a file, on one of those devices, however it the device ID of what it binds. No flag turns this off. For each operand,
reaches it: as an operand, below one, or through a bind mount, which keeps the directory or file it leaves out this way, `scan` prints a line on stderr
device ID of what it binds. The content phase never reads a file on one of naming it and the type of its filesystem, as it prints a warning, so it never
them either. No flag turns this off. For each operand, directory or file it lands inside a progress display:
leaves out this way, `scan` prints a line on stderr naming it and the type of
its filesystem, as it prints a warning, so it never lands inside a progress
display:
``` ```
scan: skipped /dev/shm (tmpfs, not a persistent filesystem) scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
@@ -1125,22 +1086,13 @@ entrypoints are:
- `script/install-precommit` — install the git pre-commit hook, as - `script/install-precommit` — install the git pre-commit hook, as
`.git/hooks/pre-commit`, that runs `script/precommit`. `.git/hooks/pre-commit`, that runs `script/precommit`.
Every `docker build` in `script/` passes a new `CACHEBUST` build argument. In Every `docker build` in `script/` passes a new `CACHEBUST` build argument, which
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps every stage of the `Dockerfile` declares first, so every `RUN` step runs again
that install system packages and download the Go modules, or right after `FROM` each time. On an unchanged tree Docker would otherwise serve the gate steps from
in a stage with neither, and every step that lints, tests or compiles comes its cache, and the build would pass having run no test and no linter. Go's build
after it, so every `RUN` step after it runs again each time. On an unchanged cache is kept between builds on the steps that compile, so only what changed is
tree Docker would otherwise serve the gate steps from its cache, and the build compiled again; the tests run with `-count=1`, so none reports a stored result.
would pass having run no test and no linter. The layers above that line stay in Every run downloads the Go modules again and needs the network.
Docker's build cache until the base image, their own lines, `go.mod` or `go.sum`
change, or until that cache is cleared; only then are the packages installed and
the modules downloaded again. Go's build cache is kept between builds on the
steps that compile, so only what changed is compiled again; the tests run with
`-count=1`, so none reports a stored result. `golangci-lint run` also keeps
golangci-lint's cache, and takes a package's findings from it only when the
package's files and those of the packages it imports, `go.mod`, the linters and
their settings in `.golangci.yml`, and the golangci-lint version are all
unchanged.
## Build ## Build
@@ -1204,6 +1156,10 @@ All of the following, run in this directory, must pass:
./sfdupes report ./sfdupes report
``` ```
(The database lives in a temp directory of its own: inside `$d`, the scan
would record it, and its empty lock file would join the `empty1`/`empty2`
group.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
group (two dupe rows, `first` is the lexicographically smallest path); group (two dupe rows, `first` is the lexicographically smallest path);
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/ `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
+34 -67
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-10-18 last_modified: 2026-10-15
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -159,11 +159,8 @@ style conventions are in separate documents:
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own `TMPDIR`, so the lock is org, in both directions. A container has its own golangci-lint cache, its
out of reach. The lint phase does keep golangci-lint's cache between check own `TMPDIR` and a digest-pinned binary, so neither is reachable.
builds (below), but it lints every checkout at `/src`, and a finding is
taken from that cache only for the same files, settings and golangci-lint
version.
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker - **Any build that runs checks is given a new `CACHEBUST` value.** Docker
invalidates a `COPY` layer only when the copied content changes, so on an invalidates a `COPY` layer only when the copied content changes, so on an
@@ -173,21 +170,15 @@ style conventions are in separate documents:
id, so no two builds share a value, not even the three in one id, so no two builds share a value, not even the three in one
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and `script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
`script/docker` are the four, and there is no fifth — `script/check` runs the `script/docker` are the four, and there is no fifth — `script/check` runs the
two gate phases and `script/fmt-check`, and builds no image of its own. In two gate phases and `script/fmt-check`, and builds no image of its own. Every
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after
that install system packages and download Go's modules, or right after `FROM` `FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
in a stage with neither, and every step that lints, tests or compiles comes are compared by content and need nothing. A stage without the `ARG` line is
after it. A step that installs anything else as well, such as a served from cache on an unchanged tree, so a repository re-vendors the scripts
`script/bootstrap` that also installs node packages, comes after it too. A new and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
value reruns every `RUN` step after that line; `COPY` steps are compared by it reruns every step too, but it also empties every cache mount, and Go's
content and need nothing. Docker keeps the layers above the line until the build cache, which the canonical Go `Dockerfile` below mounts so that compiled
base image, their own lines or the files they copy change. A stage without the packages survive between check builds, would start empty every time. A bare
`ARG` line is served from cache on an unchanged tree, so a repository
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
not pass `--no-cache`: it reruns every step too, but it also builds the layers
above the line again and empties every cache mount, so Go's build cache and
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
survive between check builds, would start empty every time. A bare
`docker build .` is not evidence that anything ran: a sub-second build `docker build .` is not evidence that anything ran: a sub-second build
reporting success is a cache hit, not a result. Never invalidate by pruning — reporting success is a cache hit, not a result. Never invalidate by pruning —
`docker builder prune` and friends destroy a build cache shared with every `docker builder prune` and friends destroy a build cache shared with every
@@ -203,25 +194,19 @@ style conventions are in separate documents:
`Dockerfile`: `Dockerfile`:
```dockerfile ```dockerfile
# In every stage ARG CACHEBUST comes right after the steps that install # Every stage declares ARG CACHEBUST first: script/ passes a new value
# system packages and download Go's modules, or right after FROM in a # on every build, which reruns every RUN step after it. The steps that
# stage with neither, and every step that lints, tests or compiles # compile mount Go's build cache, which survives from build to build.
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build.
# Lint phase # Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD # golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint FROM golangci/golangci-lint@sha256:... AS lint
ARG CACHEBUST
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./... golangci-lint run --config .golangci.yml ./...
# Test phase. -race needs cgo and so a C compiler, which the Debian Go # Test phase. -race needs cgo and so a C compiler, which the Debian Go
@@ -229,10 +214,10 @@ style conventions are in separate documents:
# cache also holds test results, which go test would otherwise replay. # cache also holds test results, which go test would otherwise replay.
# golang:1.x, YYYY-MM-DD # golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test FROM golang@sha256:... AS test
ARG CACHEBUST
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -cover ./... || \ go test -count=1 -timeout 90s -race -cover ./... || \
@@ -240,19 +225,19 @@ style conventions are in separate documents:
go test -count=1 -timeout 90s -race -v ./...; exit 1; } go test -count=1 -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies # Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so nothing after them can # are what make BuildKit build them first, so this stage cannot run
# run unless lint and test passed. # unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD # golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder FROM golang@sha256:... AS builder
RUN apk add --no-cache git
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses. # A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . . COPY . .
# The VERSION build arg when one is given, otherwise # The VERSION build arg when one is given, otherwise
@@ -289,26 +274,16 @@ style conventions are in separate documents:
- Keep the runtime stage last, and if you add a stage after it, give it the - Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain same two copies. A plain `docker build .` builds the last stage's chain
and nothing else. and nothing else.
- Check builds keep these, and none of them can hand back a stale result: - Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
- The layers above `ARG CACHEBUST`: the system packages and Go's it for root), and only on the steps that compile: `golangci-lint run`,
modules. Docker installs the packages again when the base image or the both `go test` runs and `go build`. It keeps compiled packages between
install line changes, and downloads the modules again when `go.mod` or check builds, and Go reuses one only when all its inputs are unchanged, so
`go.sum` changes. Go checks every module against `go.sum` as it it saves the compile without skipping a check. Nothing else is kept:
downloads it, and no later build can change a kept layer. The modules golangci-lint's own cache would replay stored findings in place of a lint
are kept in a layer rather than a cache mount because a cache mount is run, and Go builds from whatever files a kept module cache holds without
shared by every build on the host, and Go builds from whatever files checking them again (only `go mod verify` does), so a file changed there
it holds without checking them again (only `go mod verify` does), so a would go unnoticed. Go's build cache also holds test results, which is why
file changed there would go unnoticed. both `go test` runs carry `-count=1`.
- Go's build cache (`/root/.cache/go-build`, where Go keeps it for
root), mounted on the steps that compile: `golangci-lint run`, both
`go test` runs and `go build`. Go reuses a compiled package only when
all its inputs are unchanged. The cache also holds test results, which
is why both `go test` runs carry `-count=1`.
- golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned
image keeps it), mounted on `golangci-lint run`. golangci-lint reuses
a package's findings only when the package's files and those of every
package it imports, `go.mod`, the linters and their settings in
`.golangci.yml`, and the golangci-lint version are all unchanged.
- If the project uses `//go:embed` directives that reference build artifacts - If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must (e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
@@ -317,20 +292,12 @@ style conventions are in separate documents:
in the lint phase. The `golangci/golangci-lint` image is Debian-based and in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package (`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them. The test phase, lists in the same `RUN`, so the layer does not keep them:
on the Debian Go image, installs the libraries its tests need the same
way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so
Docker keeps it until the base image or the install line changes:
```dockerfile ```dockerfile
FROM golangci/golangci-lint@sha256:... AS lint
RUN apt-get update \ RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \ && apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
``` ```
- `.dockerignore` lets `.git` into the build context. It keeps out every git - `.dockerignore` lets `.git` into the build context. It keeps out every git
+18 -57
View File
@@ -29,55 +29,20 @@
# Completed Steps # Completed Steps
- re-vendor `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` from - `scan` never records anything that is not on a persistent filesystem (`tmpfs`,
`sneak/prompts` `next` at `0653579`: in the `lint`, `test` and `builder` `proc`, `sysfs` and the other kernel or memory-backed types), matched by the
stages of the `Dockerfile` the module download, and in `builder` the device ID of each such mount point, listed as the scan starts from
system-package install, now come before `ARG CACHEBUST`, so Docker keeps them `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
until the base image, the install line, `go.mod` or `go.sum` changes, and however the walk reaches it; no flag turns that off. Below an operand, its
every step that lints, tests or compiles still comes after it; walk does not enter the default excludes either, so `sfdupes scan /` skips the
`golangci-lint run` also keeps golangci-lint's cache. README says the same kernel's view of the running system and the temporary directories: on Linux
(2026-10-18, https://git.eeqj.de/sneak/sfdupes/issues/109) `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`, on macOS `/dev`,
`/private/tmp`, `/private/var/tmp`, `/private/var/vm`, `/System/Volumes/VM`
- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so and `/System/Volumes/Data`, and a Windows list, not used yet. An operand at or
`go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has inside a default exclude is still walked. The scan prints a line for each path
passed the build; README "Install" says so, says what `@main` and `v0.0.1` it leaves out, without counting it as skipped, and the records there count as
give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the outside the scanned operands, as for `-x`. `--no-default-excludes` turns off
early build without a database. When the linker set no version, the default excludes only, and `scan --help` lists them (2026-10-17,
`sfdupes --version` prints the main module's version that Go recorded in the
binary, the tag or the pseudo-version of a `go install` build, and `dev` only
when neither is known (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/116)
- `scan` creates the database's directory where the database path leads: the
path with its last element cut off as text, never cleaned, so a `..` after a
symlinked directory is followed through the symlink, as when the file is
opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
symlink to `/data/x`, it creates `/data/new` and the database in it, where it
used to create `/srv/new` and then fail (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/119)
- `scan` never records its own database: once the database and its `.lock` file
are open, it notes the device and inode of the database file, its `-wal` and
`-shm` files and its `.lock` file, and the walk passes over a file with one of
them wherever the operands reach it, through a symlinked directory included,
printing no line and counting nothing. A record an earlier scan left for one
of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
- `scan` never records or reads anything that is not on a persistent filesystem
(`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
matched by the device ID of each such filesystem, listed as the scan starts
from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
however the walk or the content phase reaches it; no flag turns that off.
Below an operand, its walk does not enter the default excludes either, so
`sfdupes scan /` skips the kernel's view of the running system and the
temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
`/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
`/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
Windows list, not used yet. An operand at or inside a default exclude is still
walked. The scan prints a line for each path it leaves out, without counting
it as skipped, and the records there count as outside the scanned operands, as
for `-x`. `--no-default-excludes` turns off the default excludes only, and
`scan --help` lists them (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/124) https://git.eeqj.de/sneak/sfdupes/issues/124)
- the rate a progress display shows covers the items completed over the last 5 - the rate a progress display shows covers the items completed over the last 5
@@ -163,11 +128,7 @@
and the build stage installs `git` and `make` itself; a new push cancels the and the build stage installs `git` and `make` itself; a new push cancels the
workflow's older run on the same branch, and a run stops after 20 minutes; workflow's older run on the same branch, and a run stops after 20 minutes;
`.claude/settings.json` is deleted (2026-10-15, `.claude/settings.json` is deleted (2026-10-15,
https://git.eeqj.de/sneak/sfdupes/issues/95). Since https://git.eeqj.de/sneak/sfdupes/issues/95)
https://git.eeqj.de/sneak/sfdupes/issues/109 the module download in `lint`,
`test` and the build stage, and the install of `git` and `make` in the build
stage, come before `ARG CACHEBUST`, and Docker keeps them between builds;
every step that lints, tests or compiles still runs on every build
- with `-x`, an operand under another operand is walked on its own instead of - with `-x`, an operand under another operand is walked on its own instead of
being dropped when a directory from it (from its directory, for a regular being dropped when a directory from it (from its directory, for a regular
@@ -368,9 +329,9 @@
a `CHECK_EPOCH` build argument; since a `CHECK_EPOCH` build argument; since
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/` https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile` passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile`
declares before every step that lints, tests or compiles. Run as root, the declares first. Run as root, the tests fail
tests fail `TestScanHardlinkRunFailsTogether`, because root reads through the `TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)`
`chmod(0)` the test relies on, so the `test` phase runs them as `nobody` the test relies on, so the `test` phase runs them as `nobody`
- check the installed golangci-lint version in `script/bootstrap` instead of - check the installed golangci-lint version in `script/bootstrap` instead of
only its presence (2026-08-09, branch `bootstrap-version-check`, closes only its presence (2026-08-09, branch `bootstrap-version-check`, closes
https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
+8 -60
View File
@@ -109,8 +109,10 @@ func databasePath() (string, string, error) {
} }
// The working directory as the kernel reports it, with no symlink // The working directory as the kernel reports it, with no symlink
// in it, as README "Database" says. os.Getwd may return PWD // in it. os.Getwd may return PWD instead, which can name it through
// instead, which can name it through a symlink. // a symlink; a ".." after that symlink then leads somewhere else
// once the path is cleaned, as scan cleans it to create the
// database's directory.
wd, err := unix.Getwd() wd, err := unix.Getwd()
if err != nil { if err != nil {
return "", "", fmt.Errorf("resolve database path %s: %w", path, err) return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
@@ -123,24 +125,6 @@ func databasePath() (string, string, error) {
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
} }
// createDatabaseDirectory creates the directory the database at path
// goes in, and any directory above it, if missing. path is absolute, as
// databasePath returns it. The directory is path with its last element
// cut off as text, by filepath.Split. filepath.Dir would clean it,
// dropping each ".." together with the directory name before it; when
// that name is a symlink, the cleaned path names another directory than
// the one the system reaches when it opens path.
func createDatabaseDirectory(path string) error {
dir, _ := filepath.Split(path)
err := os.MkdirAll(dir, dbDirPerm)
if err != nil {
return fmt.Errorf("create database directory: %w", err)
}
return nil
}
// scanParams are the connection parameters for scan: read-write, with // scanParams are the connection parameters for scan: read-write, with
// WAL journaling and a busy timeout, so a report can run while a cron // WAL journaling and a busy timeout, so a report can run while a cron
// scan is in progress. closeScanDatabase leaves WAL mode again. // scan is in progress. closeScanDatabase leaves WAL mode again.
@@ -195,9 +179,9 @@ func openDB(path, params string) (*sql.DB, error) {
// that deleted it would let the next scan lock a new file while // that deleted it would let the next scan lock a new file while
// another still holds the old one. // another still holds the old one.
func lockScanDatabase(path string) (*os.File, error) { func lockScanDatabase(path string) (*os.File, error) {
err := createDatabaseDirectory(path) err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("create database directory: %w", err)
} }
// Like SQLite, this follows a symlink at path and creates its // Like SQLite, this follows a symlink at path and creates its
@@ -241,9 +225,9 @@ func lockScanDatabase(path string) (*os.File, error) {
// openScanDatabase opens the database for the scan subcommand, creating // openScanDatabase opens the database for the scan subcommand, creating
// the file, its parent directory, and the schema as needed. // the file, its parent directory, and the schema as needed.
func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) { func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
err := createDatabaseDirectory(path) err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("create database directory: %w", err)
} }
db, err := openDB(path, scanParams) db, err := openDB(path, scanParams)
@@ -261,42 +245,6 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
return db, nil return db, nil
} }
// databaseFiles returns, by device and inode, the files of the database
// at path that the scan keeps open: the database file, its -wal and
// -shm files and its .lock file, each named from the database's path
// with symlinks resolved, as SQLite and lockScanDatabase name them. The
// scan calls it once the database and the lock file are open, so that
// its walk passes over them however it reaches them, through a
// symlinked directory included. A file that does not exist then is left
// out, since no walk can reach it. There is no -journal file while the
// database is in WAL mode.
func databaseFiles(path string) ([]fileRec, error) {
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return nil, fmt.Errorf("resolve database path: %w", err)
}
var files []fileRec
for _, p := range []string{
resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
} {
fi, err := os.Lstat(p)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("database: %w", err)
}
dev, ino := inodeOfInfo(fi)
files = append(files, fileRec{path: p, dev: dev, ino: ino})
}
return files, nil
}
// closeScanDatabase switches the database at path from WAL back to // closeScanDatabase switches the database at path from WAL back to
// rollback-journal mode and closes it. Out of WAL mode the database // rollback-journal mode and closes it. Out of WAL mode the database
// file alone holds the whole database, so a reader needs no -wal or // file alone holds the whole database, so a reader needs no -wal or
-111
View File
@@ -5,7 +5,6 @@ import (
"database/sql" "database/sql"
"errors" "errors"
"fmt" "fmt"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"slices" "slices"
@@ -119,116 +118,6 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
} }
} }
// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
// through a symlink to a directory, then "..", to a missing directory.
// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
// places scan creates the database's directory, must each create the
// database there, and nothing at srv/new, where the path leads once
// cleaned as text.
func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
t.Parallel()
creates := map[string]func(ctx context.Context, path string) error{
"lockScanDatabase": func(_ context.Context, path string) error {
lock, err := lockScanDatabase(path)
if err == nil {
_ = lock.Close()
}
return err
},
"openScanDatabase": func(ctx context.Context, path string) error {
db, err := openScanDatabase(ctx, path)
if err == nil {
_ = db.Close()
}
return err
},
}
for name, create := range creates {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Symlink(filepath.Join(dir, "data", "x"),
filepath.Join(dir, "srv", "link"))
if err != nil {
t.Fatal(err)
}
err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
if err != nil {
t.Fatalf("%s: %v", name, err)
}
_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
if err != nil {
t.Errorf("database the path leads to: %v", err)
}
_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("srv/new: stat error %v, want it missing", err)
}
})
}
}
// TestDatabaseFiles opens a database through a symlink to its file:
// databaseFiles finds the database file and its -wal and -shm files
// beside the symlink's target, where SQLite keeps them, and leaves out
// the .lock file, which only a scan creates.
func TestDatabaseFiles(t *testing.T) {
t.Parallel()
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
db, err := openScanDatabase(t.Context(), link)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
files, err := databaseFiles(link)
if err != nil {
t.Fatal(err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.path)
}
want := []string{path, path + "-wal", path + "-shm"}
if !slices.Equal(got, want) {
t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
}
}
func TestOpenDatabaseUnversionedForeign(t *testing.T) { func TestOpenDatabaseUnversionedForeign(t *testing.T) {
t.Parallel() t.Parallel()
+47 -33
View File
@@ -3,8 +3,8 @@ package main
// This file holds two rules for the walk (README "Rules for the walk"): // This file holds two rules for the walk (README "Rules for the walk"):
// the default excludes, the paths a scan's walk does not enter below an // the default excludes, the paths a scan's walk does not enter below an
// operand unless scan is given --no-default-excludes, and the // operand unless scan is given --no-default-excludes, and the
// filesystems that are not persistent, on which the scan never records // filesystems that are not persistent, on which the walk never records
// or reads anything. // anything.
import ( import (
"bufio" "bufio"
@@ -15,8 +15,6 @@ import (
"path/filepath" "path/filepath"
"slices" "slices"
"strings" "strings"
"golang.org/x/sys/unix"
) )
// The systems that have default excludes, as runtime.GOOS names them. // The systems that have default excludes, as runtime.GOOS names them.
@@ -27,11 +25,11 @@ const (
) )
// The fields of a line of mountInfoPath that readMountInfo reads, // The fields of a line of mountInfoPath that readMountInfo reads,
// counted from 0 (see proc(5)): the device ID, as major:minor, is the // counted from 0 (see proc(5)): the mount point is the fifth field, and
// third field, and after the sixth come optional fields, then a field // after the sixth come optional fields, then a field holding "-", then
// holding "-", then the filesystem type. // the filesystem type.
const ( const (
deviceField = 2 mountPointField = 4
optionalFieldsStart = 6 optionalFieldsStart = 6
) )
@@ -43,17 +41,14 @@ const typesPerHelpLine = 6
// readMountInfo reads. // readMountInfo reads.
var errMountInfoLine = errors.New("unexpected line") var errMountInfoLine = errors.New("unexpected line")
// walkRules holds the rules for the walk that a scan sets up as it // walkRules holds the two rules for the walk that a scan sets up as it
// starts. excludes are the default excludes, or none with // starts. excludes are the default excludes, or none with
// --no-default-excludes. nonPersistent maps the device ID of every // --no-default-excludes. nonPersistent maps the device ID of every
// mounted filesystem that is not persistent to its type; the scan // mounted filesystem that is not persistent to its type; the walk
// records and reads nothing on one of those, whatever the flags. // records nothing on one of those, whatever the flags.
// databaseFiles are the scan's own database files, which the walk
// passes over (see databaseFiles in db.go).
type walkRules struct { type walkRules struct {
excludes []string excludes []string
nonPersistent map[uint64]string nonPersistent map[uint64]string
databaseFiles []fileRec
} }
// defaultExcludes returns the default excludes of the system goos // defaultExcludes returns the default excludes of the system goos
@@ -110,6 +105,32 @@ func nonPersistentFilesystemTypes() []string {
} }
} }
// nonPersistentFilesystems returns the device ID of every mounted
// filesystem that is not persistent, with its type, read with lstat on
// its mount point. mountedFilesystems lists the mounts. A mount point
// that lstat cannot reach is left out.
func nonPersistentFilesystems() (map[uint64]string, error) {
mounts, err := mountedFilesystems()
if err != nil {
return nil, err
}
types := nonPersistentFilesystemTypes()
devices := make(map[uint64]string)
for mountPoint, fsType := range mounts {
if !slices.Contains(types, fsType) {
continue
}
if dev, ok := deviceOfPath(mountPoint); ok {
devices[dev] = fsType
}
}
return devices, nil
}
// nonPersistentLine returns the line the scan prints for path when fi, // nonPersistentLine returns the line the scan prints for path when fi,
// its lstat, puts it on a filesystem in rules.nonPersistent, and "" // its lstat, puts it on a filesystem in rules.nonPersistent, and ""
// when it is on any other. // when it is on any other.
@@ -129,13 +150,16 @@ func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
} }
// readMountInfo reads mountinfo, laid out as mountInfoPath is, and // readMountInfo reads mountinfo, laid out as mountInfoPath is, and
// returns the device ID of every mounted filesystem that is not // returns every mount point with the type of its filesystem. Where one
// persistent, with its type. The device ID is the one each line gives, // filesystem is mounted over another, a later line names the one on
// not that of the file now at its mount point, so a mount that a later // top, so the last line for a mount point decides.
// mount hides never adds the device of another filesystem. func readMountInfo(mountinfo io.Reader) (map[string]string, error) {
func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) { // The kernel writes a space, tab, newline or backslash in a mount
types := nonPersistentFilesystemTypes() // point as an octal escape.
devices := make(map[uint64]string) unescape := strings.NewReplacer(
`\040`, " ", `\011`, "\t", `\012`, "\n", `\134`, `\`)
mounts := make(map[string]string)
lines := bufio.NewScanner(mountinfo) lines := bufio.NewScanner(mountinfo)
for lines.Scan() { for lines.Scan() {
@@ -146,20 +170,10 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text()) return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
} }
var major, minor uint32 mounts[unescape.Replace(fields[mountPointField])] = fields[sep+1]
_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
if err != nil {
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
}
fsType := fields[sep+1]
if slices.Contains(types, fsType) {
devices[unix.Mkdev(major, minor)] = fsType
}
} }
return devices, lines.Err() return mounts, lines.Err()
} }
// defaultExcludesHelp describes, for the scan command's help, the // defaultExcludesHelp describes, for the scan command's help, the
+6 -26
View File
@@ -2,19 +2,12 @@
package main package main
import ( import "golang.org/x/sys/unix"
"slices"
"golang.org/x/sys/unix" // mountedFilesystems returns every mount point with the type of its
) // filesystem, from the system's list of mounted filesystems. Where one
// filesystem is mounted over another, the later one in the list counts.
// nonPersistentFilesystems returns the device ID of every mounted func mountedFilesystems() (map[string]string, error) {
// filesystem that is not persistent, with its type, from the system's
// list of mounted filesystems. The device ID is read with lstat on the
// mount point. Where one filesystem is mounted over another on the same
// mount point, the later one in the list counts. A mount point that
// lstat cannot reach is left out.
func nonPersistentFilesystems() (map[uint64]string, error) {
// The first call counts the filesystems, the second lists them. // The first call counts the filesystems, the second lists them.
n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT) n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
if err != nil { if err != nil {
@@ -35,18 +28,5 @@ func nonPersistentFilesystems() (map[uint64]string, error) {
unix.ByteSliceToString(st.Fstypename[:]) unix.ByteSliceToString(st.Fstypename[:])
} }
types := nonPersistentFilesystemTypes() return mounts, nil
devices := make(map[uint64]string)
for mountPoint, fsType := range mounts {
if !slices.Contains(types, fsType) {
continue
}
if dev, ok := deviceOfPath(mountPoint); ok {
devices[dev] = fsType
}
}
return devices, nil
} }
+7 -7
View File
@@ -12,12 +12,12 @@ import (
// process sees. // process sees.
const mountInfoPath = "/proc/self/mountinfo" const mountInfoPath = "/proc/self/mountinfo"
// nonPersistentFilesystems returns the device ID of every mounted // mountedFilesystems returns every mount point with the type of its
// filesystem that is not persistent, with its type: on Linux read from // filesystem: on Linux read from mountInfoPath (see readMountInfo), and
// mountInfoPath (see readMountInfo), and on any other system none. // on any other system none.
func nonPersistentFilesystems() (map[uint64]string, error) { func mountedFilesystems() (map[string]string, error) {
if runtime.GOOS != goosLinux { if runtime.GOOS != goosLinux {
return map[uint64]string{}, nil return map[string]string{}, nil
} }
f, err := os.Open(mountInfoPath) f, err := os.Open(mountInfoPath)
@@ -27,10 +27,10 @@ func nonPersistentFilesystems() (map[uint64]string, error) {
defer func() { _ = f.Close() }() defer func() { _ = f.Close() }()
devices, err := readMountInfo(f) mounts, err := readMountInfo(f)
if err != nil { if err != nil {
return nil, fmt.Errorf("read %s: %w", mountInfoPath, err) return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
} }
return devices, nil return mounts, nil
} }
+14 -67
View File
@@ -10,8 +10,6 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"golang.org/x/sys/unix"
) )
func TestDefaultExcludes(t *testing.T) { func TestDefaultExcludes(t *testing.T) {
@@ -63,17 +61,16 @@ func TestIsDefaultExclude(t *testing.T) {
func TestReadMountInfo(t *testing.T) { func TestReadMountInfo(t *testing.T) {
t.Parallel() t.Parallel()
// The second-to-last line has no optional fields. The tmpfs on // The second-to-last line has no optional fields. /mnt/over has an
// /data/a/b is hidden by the root filesystem's /srv, mounted over // ext4 filesystem mounted over its tmpfs.
// /data/a after it: the tmpfs still gives its own device, and the
// root filesystem's device is not listed.
mountinfo := strings.Join([]string{ mountinfo := strings.Join([]string{
"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw", "22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw", "23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw", "24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw", `25 22 0:23 / /mnt/ram\040disk rw shared:2 - tmpfs none rw`,
"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw", "26 22 0:24 / /mnt/over rw shared:3 - tmpfs none rw",
"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw", "27 26 8:2 / /mnt/over rw - ext4 /dev/sda2 rw",
"28 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
}, "\n") }, "\n")
got, err := readMountInfo(strings.NewReader(mountinfo)) got, err := readMountInfo(strings.NewReader(mountinfo))
@@ -81,23 +78,18 @@ func TestReadMountInfo(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
want := map[uint64]string{ want := map[string]string{
unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs", "/": "ext4", "/proc": "proc", "/sys": "sysfs",
unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs", "/mnt/ram disk": "tmpfs", "/mnt/over": "ext4", "/srv/auto": "autofs",
} }
if !maps.Equal(got, want) { if !maps.Equal(got, want) {
t.Errorf("readMountInfo = %v, want %v", got, want) t.Errorf("readMountInfo = %q, want %q", got, want)
} }
for _, line := range []string{ _, err = readMountInfo(strings.NewReader("22 1 8:1 / / rw,relatime\n"))
"22 1 8:1 / / rw,relatime\n", if !errors.Is(err, errMountInfoLine) {
"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n", t.Errorf("readMountInfo of a line without a type = %v, want %v",
} { err, errMountInfoLine)
_, err = readMountInfo(strings.NewReader(line))
if !errors.Is(err, errMountInfoLine) {
t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
errMountInfoLine)
}
} }
} }
@@ -450,51 +442,6 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
} }
} }
// TestScanContentSkipsNonPersistent checks that the content phase does
// not read a file on a filesystem that is not persistent, though an
// earlier scan left a record for it: it is treated as a changed file,
// so its record is left as it is and its identical copy in another
// tree, having no other match, is not read either.
func TestScanContentSkipsNonPersistent(t *testing.T) {
t.Parallel()
shm, _, rules := shmTempDir(t)
db := openTestDB(t)
// A scan that, without the list of filesystems, records a with a
// head and tail but no content hash.
a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
syncTree(t, db, shm)
before := recordByPath(t, dbRecords(t, db), a)
if before.head == "" || before.content != "" {
t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
}
b := sparseFile(t, t.TempDir(), "b", headTailMin)
st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
rules)
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
if r := recordByPath(t, recs, a); r != before {
t.Errorf("record = %+v, want it left as %+v", r, before)
}
if r := recordByPath(t, recs, b); r.content != "" {
t.Errorf("%s: content = %q, want none: its only match is on %s",
b, r.content, shm)
}
}
// TestWalkLeavesOutNonPersistent starts the walk itself in a directory // TestWalkLeavesOutNonPersistent starts the walk itself in a directory
// on a filesystem that is not persistent, which a scan never does: it // on a filesystem that is not persistent, which a scan never does: it
// drops such an operand first. Below a directory on a persistent // drops such an operand first. Below a directory on a persistent
-2
View File
@@ -24,5 +24,3 @@ require (
modernc.org/mathutil v1.7.1 // indirect modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect modernc.org/memory v1.11.0 // indirect
) )
retract v0.0.1 // early build without a database; does not work as the README describes
+3 -29
View File
@@ -28,7 +28,6 @@ import (
"io" "io"
"os" "os"
"runtime" "runtime"
"runtime/debug"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@@ -59,32 +58,10 @@ var errNoSubcommand = errors.New("no subcommand")
var errWorkersBelowOne = errors.New("--workers must be at least 1") var errWorkersBelowOne = errors.New("--workers must be at least 1")
// Version is the build version, injected at link time via -ldflags // Version is the build version, injected at link time via -ldflags
// (see the Makefile); empty when the linker set none, as for a binary // (see the Makefile); "dev" for a plain go build.
// built by go install. buildVersion turns it into what --version prints.
// //
//nolint:gochecknoglobals // written only by the linker //nolint:gochecknoglobals // written only by the linker
var Version string var Version = "dev"
// buildVersion returns the version that sfdupes --version prints
// (README "Install"): the version the linker set, when it set one;
// otherwise the main module's version that Go recorded in the binary,
// a tag such as v0.0.2 or the pseudo-version of an @main build; and
// "dev" when neither is known. info is what debug.ReadBuildInfo
// returns, nil when the binary carries none. Go records "(devel)" for
// a build that has no version, such as one inside a copy of the source
// without git.
func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
if linkerVersion != "" {
return linkerVersion
}
if info == nil || info.Main.Version == "" ||
info.Main.Version == "(devel)" {
return "dev"
}
return info.Main.Version
}
func main() { func main() {
// Once the reader of a stdout pipe has gone, as in "sfdupes report | // Once the reader of a stdout pipe has gone, as in "sfdupes report |
@@ -139,10 +116,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
var showVersion bool var showVersion bool
printVersion := runE(func(context.Context, []string) error { printVersion := runE(func(context.Context, []string) error {
info, _ := debug.ReadBuildInfo() _, err := fmt.Fprintf(stdout, "sfdupes %s\n", Version)
_, err := fmt.Fprintf(stdout, "sfdupes %s\n",
buildVersion(Version, info))
if err != nil { if err != nil {
return fmt.Errorf("write stdout: %w", err) return fmt.Errorf("write stdout: %w", err)
} }
+1 -38
View File
@@ -9,7 +9,6 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"runtime/debug"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -715,8 +714,6 @@ func TestRunVersion(t *testing.T) {
// README §Subcommands: the version is one line on stdout, with // README §Subcommands: the version is one line on stdout, with
// nothing on stderr, and exits 0. // nothing on stderr, and exits 0.
info, _ := debug.ReadBuildInfo()
for _, arg := range []string{"--version", "-v"} { for _, arg := range []string{"--version", "-v"} {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -725,7 +722,7 @@ func TestRunVersion(t *testing.T) {
t.Errorf("run(%s) = %d, want %d", arg, code, exitOK) t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
} }
want := "sfdupes " + buildVersion(Version, info) + "\n" want := "sfdupes " + Version + "\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("run(%s) stdout = %q, want %q", arg, got, want) t.Errorf("run(%s) stdout = %q, want %q", arg, got, want)
} }
@@ -736,40 +733,6 @@ func TestRunVersion(t *testing.T) {
} }
} }
func TestBuildVersion(t *testing.T) {
t.Parallel()
// README "Install": the linker's version wins; otherwise the main
// module's version that Go recorded; "dev" when neither is known.
tagged := &debug.BuildInfo{Main: debug.Module{Version: "v0.0.2"}}
pseudo := &debug.BuildInfo{Main: debug.Module{
Version: "v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1",
}}
devel := &debug.BuildInfo{Main: debug.Module{Version: "(devel)"}}
cases := []struct {
name string
linker string
info *debug.BuildInfo
want string
}{
{"linker set", "v0.0.2-3-g4e4a94d", tagged, "v0.0.2-3-g4e4a94d"},
{"linker set, no build info", "4e4a94d", nil, "4e4a94d"},
{"build info with a tag", "", tagged, "v0.0.2"},
{"build info with a pseudo-version", "", pseudo,
"v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1"},
{"(devel)", "", devel, "dev"},
{"no build info", "", nil, "dev"},
}
for _, c := range cases {
if got := buildVersion(c.linker, c.info); got != c.want {
t.Errorf("%s: buildVersion(%q, ...) = %q, want %q",
c.name, c.linker, got, c.want)
}
}
}
func TestRunVersionWriteFailureIsFatal(t *testing.T) { func TestRunVersionWriteFailureIsFatal(t *testing.T) {
t.Parallel() t.Parallel()
+30 -64
View File
@@ -118,8 +118,7 @@ func scanPhase(name string) string {
// rejects anything less. Before the lock is taken, the scan lists the // rejects anything less. Before the lock is taken, the scan lists the
// filesystems that are not persistent, on which the walk records // filesystems that are not persistent, on which the walk records
// nothing; unless noDefaultExcludes is set, the walk does not enter the // nothing; unless noDefaultExcludes is set, the walk does not enter the
// default excludes either. Once the database is open, the scan notes // default excludes either.
// its files, which the walk passes over.
func runScan(ctx context.Context, roots []string, workers int, func runScan(ctx context.Context, roots []string, workers int,
oneFS, noDefaultExcludes bool, oneFS, noDefaultExcludes bool,
) error { ) error {
@@ -174,11 +173,6 @@ func runScan(ctx context.Context, roots []string, workers int,
defer closeScanDatabase(ctx, db, dbPath) defer closeScanDatabase(ctx, db, dbPath)
rules.databaseFiles, err = databaseFiles(dbPath)
if err != nil {
return err
}
st, err := syncScan(ctx, db, roots, workers, oneFS, rules) st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
if errors.Is(err, context.Canceled) { if errors.Is(err, context.Canceled) {
return interrupted(st.walked) return interrupted(st.walked)
@@ -441,7 +435,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
return err return err
} }
return s.contentPhase(ctx, workers, rules) return s.contentPhase(ctx, workers)
} }
// walkableRoots returns the operands the walk can start from: regular // walkableRoots returns the operands the walk can start from: regular
@@ -716,15 +710,6 @@ func sameInode(a, b fileRec) bool {
return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
} }
// isDatabaseFile reports whether f, a file the walk reached, is one of
// the scan's own database files in rules, which the walk passes over
// without a line and without counting it.
func isDatabaseFile(f fileRec, rules walkRules) bool {
return slices.ContainsFunc(rules.databaseFiles, func(d fileRec) bool {
return sameInode(d, f)
})
}
// hashPhase hashes every queued file with hashSignature — the head and // hashPhase hashes every queued file with hashSignature — the head and
// tail of a file of headTailMin or more, the whole file below that — // tail of a file of headTailMin or more, the whole file below that —
// committing completed records to the database in batches as results // committing completed records to the database in batches as results
@@ -893,12 +878,9 @@ func (s *scanState) updatePhase(ctx context.Context) error {
// hash phase's worker pool and their records written back in batches. A // hash phase's worker pool and their records written back in batches. A
// failed read is warned about and counted as skipped; the record keeps // failed read is warned about and counted as skipped; the record keeps
// its empty content, so it is never grouped, and a later scan tries // its empty content, so it is never grouped, and a later scan tries
// again. No file on a filesystem in rules that is not persistent is // again.
// read. func (s *scanState) contentPhase(ctx context.Context, workers int) error {
func (s *scanState) contentPhase(ctx context.Context, workers int, toRead, recs, err := s.contentCandidates(ctx)
rules walkRules,
) error {
toRead, recs, err := s.contentCandidates(ctx, rules)
if err != nil { if err != nil {
return err return err
} }
@@ -933,19 +915,18 @@ func (s *scanState) contentPhase(ctx context.Context, workers int,
// contentCandidates returns the files the content phase reads, and // contentCandidates returns the files the content phase reads, and
// their records by path. Every record contentCandidatesSQL returns has // their records by path. Every record contentCandidatesSQL returns has
// its file checked with lstat, whether or not it already has a content // its file checked with lstat, whether or not it already has a content
// hash: a file that is gone, is no longer a regular file, has changed // hash: a file that is gone, is no longer a regular file, or has
// by the walk's rule, or is on a filesystem in rules that is not // changed by the walk's rule keeps its record as it is and does not
// persistent keeps its record as it is and does not count as a match // count as a match for the others, and any other lstat error is warned
// for the others, and any other lstat error is warned about and counted // about and counted as skipped, with the same result. If such a record
// as skipped, with the same result. If such a record has no content // has no content hash, it stays out of duplicate groups; if it has one,
// hash, it stays out of duplicate groups; if it has one, it is still // it is still reported until a scan covering its own tree updates or
// reported until a scan covering its own tree updates or removes it. // removes it. The files of a group that pass and have no content hash
// The files of a group that pass and have no content hash are read only // are read only if at least minGroupSize of the group's files pass, so
// if at least minGroupSize of the group's files pass, so a group whose // a group whose other members are all stale costs no reads. Only the
// other members are all stale costs no reads. Only the records to be // records to be read are kept.
// read are kept.
func (s *scanState) contentCandidates( func (s *scanState) contentCandidates(
ctx context.Context, rules walkRules, ctx context.Context,
) ([]fileRec, map[string]scanRec, error) { ) ([]fileRec, map[string]scanRec, error) {
// The search and the checks take real time on a large database; // The search and the checks take real time on a large database;
// without a display the scan looks hung before the reads begin. // without a display the scan looks hung before the reads begin.
@@ -1003,7 +984,7 @@ func (s *scanState) contentCandidates(
first = r first = r
} }
f, ok, err := unchangedFile(r, rules) f, ok, err := unchangedFile(r)
if err != nil { if err != nil {
s.st.skipped++ s.st.skipped++
@@ -1037,10 +1018,9 @@ func (s *scanState) contentCandidates(
// unchangedFile lstats the file r names and returns it for reading if // unchangedFile lstats the file r names and returns it for reading if
// it is still the regular file r records: the same size, and an mtime // it is still the regular file r records: the same size, and an mtime
// no newer than recorded (the walk's change rule). A file that is gone, // no newer than recorded (the walk's change rule). A file that is gone
// has changed, or is on a filesystem in rules that is not persistent // or has changed reports false; any other lstat error is returned.
// reports false; any other lstat error is returned. func unchangedFile(r scanRec) (fileRec, bool, error) {
func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
fi, err := os.Lstat(r.path) fi, err := os.Lstat(r.path)
if errors.Is(err, fs.ErrNotExist) { if errors.Is(err, fs.ErrNotExist) {
return fileRec{}, false, nil return fileRec{}, false, nil
@@ -1057,12 +1037,6 @@ func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
dev, ino := inodeOfInfo(fi) dev, ino := inodeOfInfo(fi)
// The scan reads nothing on a filesystem that is not persistent,
// whatever record an earlier scan left for the file.
if _, found := rules.nonPersistent[dev]; found {
return fileRec{}, false, nil
}
return fileRec{ return fileRec{
path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino, path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
}, true, nil }, true, nil
@@ -1158,7 +1132,7 @@ func startWalk(ctx context.Context, roots []string, oneFS bool,
go func() { go func() {
initial := make([]dirJob, 0, len(roots)) initial := make([]dirJob, 0, len(roots))
for _, root := range roots { for _, root := range roots {
initial = append(initial, seedRoot(ctx, root, rules, events)...) initial = append(initial, seedRoot(ctx, root, events)...)
} }
dispatchDirs(ctx, initial, jobs, subdirs) dispatchDirs(ctx, initial, jobs, subdirs)
@@ -1212,14 +1186,13 @@ func operandWarning(root string, fi fs.FileInfo) string {
} }
// seedRoot turns one PATH operand into the walk's starting state: a // seedRoot turns one PATH operand into the walk's starting state: a
// regular-file operand is statted and emitted directly, unless it is // regular-file operand is statted and emitted directly, and a directory
// one of the scan's own database files in rules, and a directory
// operand becomes an initial job. walkableRoots has already dropped // operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since // every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records // is warned about and skipped here; it is still a root, so the records
// stored beneath it, other than those under another root, are deleted // stored beneath it, other than those under another root, are deleted
// as unverified. // as unverified.
func seedRoot(ctx context.Context, root string, rules walkRules, func seedRoot(ctx context.Context, root string,
events chan<- walkEvent, events chan<- walkEvent,
) []dirJob { ) []dirJob {
fi, err := os.Lstat(root) fi, err := os.Lstat(root)
@@ -1246,17 +1219,14 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
} }
dev, ino := inodeOfInfo(fi) dev, ino := inodeOfInfo(fi)
rec := fileRec{
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: root, path: root,
size: fi.Size(), size: fi.Size(),
mtime: fi.ModTime(), mtime: fi.ModTime(),
dev: dev, dev: dev,
ino: ino, ino: ino,
} }})
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
return nil return nil
} }
@@ -1391,8 +1361,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
// regular file) between the directory read and the lstat is warned // regular file) between the directory read and the lstat is warned
// about and skipped. A file on a filesystem in rules that is not // about and skipped. A file on a filesystem in rules that is not
// persistent, as through a bind mount of one file, is not recorded: it // persistent, as through a bind mount of one file, is not recorded: it
// gets an event naming it instead. One of the scan's own database files // gets an event naming it instead.
// in rules gets no event at all.
func emitFile(ctx context.Context, p string, e fs.DirEntry, func emitFile(ctx context.Context, p string, e fs.DirEntry,
rules walkRules, events chan<- walkEvent, rules walkRules, events chan<- walkEvent,
) { ) {
@@ -1418,17 +1387,14 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
} }
dev, ino := inodeOfInfo(info) dev, ino := inodeOfInfo(info)
rec := fileRec{
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: p, path: p,
size: info.Size(), size: info.Size(),
mtime: info.ModTime(), mtime: info.ModTime(),
dev: dev, dev: dev,
ino: ino, ino: ino,
} }})
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
} }
// subdirJob applies the descent rules to directory p: never enter // subdirJob applies the descent rules to directory p: never enter
+1 -97
View File
@@ -1171,7 +1171,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
var initial []dirJob var initial []dirJob
for _, root := range roots { for _, root := range roots {
for _, job := range seedRoot(ctx, root, walkRules{}, events) { for _, job := range seedRoot(ctx, root, events) {
job.rootDev = devices[root] job.rootDev = devices[root]
initial = append(initial, job) initial = append(initial, job)
} }
@@ -2501,99 +2501,3 @@ func TestUnderRoot(t *testing.T) {
} }
} }
} }
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
// database, reached directly and through a symlinked directory. The
// walk passes over the database file, its -wal and -shm files and its
// .lock file with no line and no count, and the record an earlier scan
// left for the .lock file is deleted. A rescan of the unchanged tree
// updates nothing, and report lists none of the database's files.
func TestRunScanPassesOverDatabase(t *testing.T) {
for _, viaSymlink := range []bool{false, true} {
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
base := t.TempDir()
tree := filepath.Join(base, "real", "tree")
operand := tree
writeFile(t, tree, "a", pattern(1, 100))
writeFile(t, tree, "b", pattern(1, 100))
writeFile(t, tree, "empty1", nil)
writeFile(t, tree, "empty2", nil)
if viaSymlink {
err := os.Symlink(filepath.Join(base, "real"),
filepath.Join(base, "link"))
if err != nil {
t.Fatal(err)
}
operand = filepath.Join(base, "link", "tree")
}
dbPath := filepath.Join(tree, "db.sqlite")
t.Setenv(databaseEnv, dbPath)
db, err := openScanDatabase(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
err = applyChanges(t.Context(), db, []scanRec{
{path: filepath.Join(operand, "db.sqlite.lock")},
}, nil, nil)
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, dbPath)
assertScanSummary(t,
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
operand)
assertScanSummary(t,
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
operand)
report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") ||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
t.Errorf("report = %q, want the empty files and none of "+
"the database's files", report)
}
})
}
}
// TestRunScanPassesOverDatabaseOperands scans the database file and its
// .lock file given as operands themselves: neither is recorded or
// counted.
func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
dbPath := testDBPath(t)
t.Setenv(databaseEnv, dbPath)
// The first scan creates both files.
scanOK(t, t.TempDir())
assertScanSummary(t,
"0 files seen (0 added, 0 updated, 0 removed, 0 unchanged)",
dbPath, dbPath+".lock")
}
// assertScanSummary scans operands and fails the test unless the
// summary line gives counts with nothing skipped, and the line naming
// the database is the only one that names one of its files.
func assertScanSummary(t *testing.T, counts string, operands ...string) {
t.Helper()
stderr := scanOK(t, operands...)
want := "scan: " + counts + ", 0 skipped\n"
if !strings.Contains(stderr, want) {
t.Errorf("scan %q: stderr = %q, want %q", operands, stderr, want)
}
if strings.Count(stderr, "db.sqlite") != 1 {
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
"database to name one of its files", operands, stderr)
}
}
+4 -5
View File
@@ -2,12 +2,11 @@
# script/cibuild: run the CI build. It bootstraps first: a CI runner # script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs # checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do. # the formatter on the host, which a pristine checkout cannot do.
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST # A new CACHEBUST value reruns every RUN step, since a cached gate phase
# line, since a cached gate phase is a check that did not run; unlike # is a check that did not run; unlike --no-cache, it keeps Go's build cache.
# --no-cache, it keeps the layers above that line and the cache mounts.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST # This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# before every step that lints, tests or compiles; a stage without that # first; a stage without that line is served from cache on an unchanged
# line is served from cache on an unchanged tree, with no warning. # tree, with no warning.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+4 -5
View File
@@ -1,12 +1,11 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST # A new CACHEBUST value reruns every RUN step, since a cached gate phase
# line, since a cached gate phase is a check that did not run; unlike # is a check that did not run; unlike --no-cache, it keeps Go's build cache.
# --no-cache, it keeps the layers above that line and the cache mounts.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST # This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# before every step that lints, tests or compiles; a stage without that # first; a stage without that line is served from cache on an unchanged
# line is served from cache on an unchanged tree, with no warning. # tree, with no warning.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+5 -7
View File
@@ -5,13 +5,11 @@
# make its answer untrustworthy. # make its answer untrustworthy.
# #
# The phase is not the last stage in the file, so it is built only when # The phase is not the last stage in the file, so it is built only when
# --target names it. A new CACHEBUST value reruns every RUN step after # --target names it. A new CACHEBUST value reruns every RUN step, since
# the ARG CACHEBUST line, since a cached lint layer is a lint that did # a cached lint layer is a lint that did not run; unlike --no-cache, it
# not run; unlike --no-cache, it keeps the layers above that line and # keeps Go's build cache. This relies on every stage of the Dockerfile
# the cache mounts. This relies on every stage of the Dockerfile # declaring ARG CACHEBUST first; a stage without that line is served
# declaring ARG CACHEBUST before every step that lints, tests or # from cache on an unchanged tree, with no warning.
# compiles; a stage without that line is served from cache on an
# unchanged tree, with no warning.
# --output type=cacheonly writes no image, since nothing uses one. # --output type=cacheonly writes no image, since nothing uses one.
set -eu set -eu
+5 -6
View File
@@ -3,12 +3,11 @@
# and this builds that phase alone, on the same terms as script/lint: # and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when # --target because a phase that is not the last stage is built only when
# named, and a new CACHEBUST value because a cached test layer is a test # named, and a new CACHEBUST value because a cached test layer is a test
# that did not run; unlike --no-cache, it keeps the layers above the ARG # that did not run; unlike --no-cache, it keeps Go's build cache. This
# CACHEBUST line and the cache mounts. This relies on every stage of the # relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
# Dockerfile declaring ARG CACHEBUST before every step that lints, tests # a stage without that line is served from cache on an unchanged tree,
# or compiles; a stage without that line is served from cache on an # with no warning. --output type=cacheonly writes no image, since
# unchanged tree, with no warning. --output type=cacheonly writes no # nothing uses one.
# image, since nothing uses one.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"