1 Commits
Author SHA1 Message Date
sneak 08323879ec scan: skip OS default excludes, and record nothing on filesystems that are not persistent (closes #124)
check / check (push) Canceled after 0s
scan never records or reads anything on a filesystem that is not
persistent (tmpfs, proc, sysfs and the like), however it reaches it; no
flag turns this off. It notes each such filesystem's device ID as it
starts: on Linux from /proc/self/mountinfo, on macOS by lstat of each
listed mount point. Below an operand the walk also skips the default
excludes, a path list per system (Linux, macOS, and Windows, not used
yet); --no-default-excludes turns off only those. A default exclude that
is itself an operand gets no skip line. The tests stop at once, saying
why, when TMPDIR is not on a persistent filesystem.

Judgement call: an unreadable mount list fails the scan, and no flag gets round it.
Judgement call: a file on such a filesystem below an operand gets the same line as a directory.
Judgement call: such an operand that another operand's walk also reaches is named twice.
Unverified: whether macOS's mount list gives the device lstat reports, so macOS keeps lstat, and a hidden mount there can still add another filesystem's device.
Unverified: the macOS code is never built or run here; bind mounts need root to test.

Model: opus-5-5
2026-10-17 01:11:21 +00:00
16 changed files with 111 additions and 584 deletions

No files matched your search

+11 -19
View File
@@ -1,23 +1,16 @@
# In every stage ARG CACHEBUST comes right after the steps that install # Every stage declares ARG CACHEBUST first: script/ passes a new value
# system packages and download Go's modules, or right after FROM in a # on every build, which reruns every RUN step after it. The steps that
# stage with neither, and every step that lints, tests or compiles # compile mount Go's build cache, which survives from build to build.
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build. The
# test phase runs as nobody and so mounts Go's build cache at
# /tmp/.cache/go-build; its own comment says why.
# Lint phase, built alone by script/lint. The tools are invoked directly # Lint phase, built alone by script/lint. The tools are invoked directly
# rather than through `make lint`, which runs docker itself and so cannot # rather than through `make lint`, which runs docker itself and so cannot
# run inside a build step. # run inside a build step.
# golangci/golangci-lint:v2.14.0, 2026-10-07 # golangci/golangci-lint:v2.14.0, 2026-10-07
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
ARG CACHEBUST
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
# The gofmt half of `make fmt-check`. gofmt's output is assigned to a # The gofmt half of `make fmt-check`. gofmt's output is assigned to a
@@ -31,7 +24,6 @@ RUN files="$(gofmt -s -l .)" && \
# Validates .golangci.yml against the schema the pinned binary embeds. # Validates .golangci.yml against the schema the pinned binary embeds.
RUN golangci-lint config verify --config .golangci.yml RUN golangci-lint config verify --config .golangci.yml
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./... golangci-lint run --config .golangci.yml ./...
# Test phase, built alone by script/test. -race needs cgo and so a C # Test phase, built alone by script/test. -race needs cgo and so a C
@@ -46,12 +38,12 @@ RUN --mount=type=cache,target=/root/.cache/go-build \
# nobody (65534) so the tests can write to it. # nobody (65534) so the tests can write to it.
# golang:1.25-trixie, 2026-10-04 # golang:1.25-trixie, 2026-10-04
FROM golang@sha256:2c4c60ef415fbfa5e90300722293bef36c5e63fae17570ce18f580af933dbd73 AS test FROM golang@sha256:2c4c60ef415fbfa5e90300722293bef36c5e63fae17570ce18f580af933dbd73 AS test
ARG CACHEBUST
USER nobody USER nobody
ENV HOME=/tmp ENV HOME=/tmp
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN --mount=type=cache,target=/tmp/.cache/go-build,uid=65534,gid=65534 \ RUN --mount=type=cache,target=/tmp/.cache/go-build,uid=65534,gid=65534 \
go test -count=1 -timeout 90s -race -cover ./... || \ go test -count=1 -timeout 90s -race -cover ./... || \
@@ -59,19 +51,19 @@ RUN --mount=type=cache,target=/tmp/.cache/go-build,uid=65534,gid=65534 \
go test -count=1 -timeout 90s -race -v ./...; exit 1; } go test -count=1 -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies are # Build stage. Nothing is wanted from either phase above; the copies are
# what make BuildKit build them first, so nothing after them can run # what make BuildKit build them first, so this stage cannot run unless
# unless lint and test passed. # lint and test passed.
# golang:1.25-alpine, 2026-07-23 # golang:1.25-alpine, 2026-07-23
FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
RUN apk add --no-cache git make
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git make
# A tar-stream context keeps the sender's file owners, which git refuses. # A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . . COPY . .
# The version stamped into the binary: the VERSION build argument when # The version stamped into the binary: the VERSION build argument when
+19 -58
View File
@@ -49,40 +49,15 @@ results are as fresh as the last completed scan.
### Install ### Install
With Go installed, this builds and installs the newest release: With Go installed, this builds and installs the current `main` branch:
```sh
go install sneak.berlin/go/sfdupes@latest
```
Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
`@main` installs the tip before it is tagged:
```sh ```sh
go install sneak.berlin/go/sfdupes@main go install sneak.berlin/go/sfdupes@main
``` ```
`v0.0.1` is an early build without a database. It does not work as this README The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. A
describes, and `go.mod` retracts it. binary installed this way reports its version as `dev`; one built from a clone
or into the Docker image carries the git tag or commit it was built from.
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
see which version an installed binary is:
```sh
sfdupes --version
go version -m "$(command -v sfdupes)"
```
`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
or for `@main` a pseudo-version ending in the first 12 characters of the
commit's hash. A binary built from a clone or into the Docker image carries the
git tag or commit it was built from instead, and `VERSION` is `dev` only when
the binary carries no version at all. `go version -m` prints what Go recorded in
the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
and earlier print `dev`, so check those with `go version -m`.
From a clone, `make build` writes the binary to `./sfdupes`: From a clone, `make build` writes the binary to `./sfdupes`:
@@ -287,10 +262,10 @@ sfdupes [command] --help
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem, and `--no-default-excludes` lets the walk enter the operand's filesystem, and `--no-default-excludes` lets the walk enter the
default excludes, a list of paths; all three are described under "`scan` mode". default excludes, a list of paths; all three are described under "`scan` mode".
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see `sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`, exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
alone or after a subcommand, prints the help text to stderr and exits 0, writing subcommand, prints the help text to stderr and exits 0, writing nothing to
nothing to stdout. stdout.
### Database ### Database
@@ -600,15 +575,6 @@ Rules for the walk:
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking - Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
them would list every file once per snapshot), not even when it is an operand; them would list every file once per snapshot), not even when it is an operand;
such an operand is dropped the same way. such an operand is dropped the same way.
- Never record the scan's own database. Once the database and its `.lock` file
are open, `scan` notes the device and inode of the database file, its `-wal`
and `-shm` files and its `.lock` file, and the walk passes over a file with
one of them, however it reaches it: as an operand, below one, or through a
symlinked directory. It prints no line for such a file and does not count it
as skipped. A record that an earlier scan left for one of them is deleted,
like the record of any file the scan no longer verifies. There is no
`-journal` file during the walk, because the database is in WAL mode then (see
"Database").
- Filesystem boundaries are crossed by default. With `-x` (long form - Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend `--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each into a directory on a different filesystem than its `PATH` operand; each
@@ -1125,22 +1091,13 @@ entrypoints are:
- `script/install-precommit` — install the git pre-commit hook, as - `script/install-precommit` — install the git pre-commit hook, as
`.git/hooks/pre-commit`, that runs `script/precommit`. `.git/hooks/pre-commit`, that runs `script/precommit`.
Every `docker build` in `script/` passes a new `CACHEBUST` build argument. In Every `docker build` in `script/` passes a new `CACHEBUST` build argument, which
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps every stage of the `Dockerfile` declares first, so every `RUN` step runs again
that install system packages and download the Go modules, or right after `FROM` each time. On an unchanged tree Docker would otherwise serve the gate steps from
in a stage with neither, and every step that lints, tests or compiles comes its cache, and the build would pass having run no test and no linter. Go's build
after it, so every `RUN` step after it runs again each time. On an unchanged cache is kept between builds on the steps that compile, so only what changed is
tree Docker would otherwise serve the gate steps from its cache, and the build compiled again; the tests run with `-count=1`, so none reports a stored result.
would pass having run no test and no linter. The layers above that line stay in Every run downloads the Go modules again and needs the network.
Docker's build cache until the base image, their own lines, `go.mod` or `go.sum`
change, or until that cache is cleared; only then are the packages installed and
the modules downloaded again. Go's build cache is kept between builds on the
steps that compile, so only what changed is compiled again; the tests run with
`-count=1`, so none reports a stored result. `golangci-lint run` also keeps
golangci-lint's cache, and takes a package's findings from it only when the
package's files and those of the packages it imports, `go.mod`, the linters and
their settings in `.golangci.yml`, and the golangci-lint version are all
unchanged.
## Build ## Build
@@ -1204,6 +1161,10 @@ All of the following, run in this directory, must pass:
./sfdupes report ./sfdupes report
``` ```
(The database lives in a temp directory of its own: inside `$d`, the scan
would record it, and its empty lock file would join the `empty1`/`empty2`
group.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
group (two dupe rows, `first` is the lexicographically smallest path); group (two dupe rows, `first` is the lexicographically smallest path);
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/ `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
+34 -67
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-10-18 last_modified: 2026-10-15
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -159,11 +159,8 @@ style conventions are in separate documents:
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own `TMPDIR`, so the lock is org, in both directions. A container has its own golangci-lint cache, its
out of reach. The lint phase does keep golangci-lint's cache between check own `TMPDIR` and a digest-pinned binary, so neither is reachable.
builds (below), but it lints every checkout at `/src`, and a finding is
taken from that cache only for the same files, settings and golangci-lint
version.
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker - **Any build that runs checks is given a new `CACHEBUST` value.** Docker
invalidates a `COPY` layer only when the copied content changes, so on an invalidates a `COPY` layer only when the copied content changes, so on an
@@ -173,21 +170,15 @@ style conventions are in separate documents:
id, so no two builds share a value, not even the three in one id, so no two builds share a value, not even the three in one
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and `script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
`script/docker` are the four, and there is no fifth — `script/check` runs the `script/docker` are the four, and there is no fifth — `script/check` runs the
two gate phases and `script/fmt-check`, and builds no image of its own. In two gate phases and `script/fmt-check`, and builds no image of its own. Every
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after
that install system packages and download Go's modules, or right after `FROM` `FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
in a stage with neither, and every step that lints, tests or compiles comes are compared by content and need nothing. A stage without the `ARG` line is
after it. A step that installs anything else as well, such as a served from cache on an unchanged tree, so a repository re-vendors the scripts
`script/bootstrap` that also installs node packages, comes after it too. A new and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
value reruns every `RUN` step after that line; `COPY` steps are compared by it reruns every step too, but it also empties every cache mount, and Go's
content and need nothing. Docker keeps the layers above the line until the build cache, which the canonical Go `Dockerfile` below mounts so that compiled
base image, their own lines or the files they copy change. A stage without the packages survive between check builds, would start empty every time. A bare
`ARG` line is served from cache on an unchanged tree, so a repository
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
not pass `--no-cache`: it reruns every step too, but it also builds the layers
above the line again and empties every cache mount, so Go's build cache and
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
survive between check builds, would start empty every time. A bare
`docker build .` is not evidence that anything ran: a sub-second build `docker build .` is not evidence that anything ran: a sub-second build
reporting success is a cache hit, not a result. Never invalidate by pruning — reporting success is a cache hit, not a result. Never invalidate by pruning —
`docker builder prune` and friends destroy a build cache shared with every `docker builder prune` and friends destroy a build cache shared with every
@@ -203,25 +194,19 @@ style conventions are in separate documents:
`Dockerfile`: `Dockerfile`:
```dockerfile ```dockerfile
# In every stage ARG CACHEBUST comes right after the steps that install # Every stage declares ARG CACHEBUST first: script/ passes a new value
# system packages and download Go's modules, or right after FROM in a # on every build, which reruns every RUN step after it. The steps that
# stage with neither, and every step that lints, tests or compiles # compile mount Go's build cache, which survives from build to build.
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build.
# Lint phase # Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD # golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint FROM golangci/golangci-lint@sha256:... AS lint
ARG CACHEBUST
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./... golangci-lint run --config .golangci.yml ./...
# Test phase. -race needs cgo and so a C compiler, which the Debian Go # Test phase. -race needs cgo and so a C compiler, which the Debian Go
@@ -229,10 +214,10 @@ style conventions are in separate documents:
# cache also holds test results, which go test would otherwise replay. # cache also holds test results, which go test would otherwise replay.
# golang:1.x, YYYY-MM-DD # golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test FROM golang@sha256:... AS test
ARG CACHEBUST
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
ARG CACHEBUST
COPY . . COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -cover ./... || \ go test -count=1 -timeout 90s -race -cover ./... || \
@@ -240,19 +225,19 @@ style conventions are in separate documents:
go test -count=1 -timeout 90s -race -v ./...; exit 1; } go test -count=1 -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies # Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so nothing after them can # are what make BuildKit build them first, so this stage cannot run
# run unless lint and test passed. # unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD # golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder FROM golang@sha256:... AS builder
RUN apk add --no-cache git
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses. # A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . . COPY . .
# The VERSION build arg when one is given, otherwise # The VERSION build arg when one is given, otherwise
@@ -289,26 +274,16 @@ style conventions are in separate documents:
- Keep the runtime stage last, and if you add a stage after it, give it the - Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain same two copies. A plain `docker build .` builds the last stage's chain
and nothing else. and nothing else.
- Check builds keep these, and none of them can hand back a stale result: - Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
- The layers above `ARG CACHEBUST`: the system packages and Go's it for root), and only on the steps that compile: `golangci-lint run`,
modules. Docker installs the packages again when the base image or the both `go test` runs and `go build`. It keeps compiled packages between
install line changes, and downloads the modules again when `go.mod` or check builds, and Go reuses one only when all its inputs are unchanged, so
`go.sum` changes. Go checks every module against `go.sum` as it it saves the compile without skipping a check. Nothing else is kept:
downloads it, and no later build can change a kept layer. The modules golangci-lint's own cache would replay stored findings in place of a lint
are kept in a layer rather than a cache mount because a cache mount is run, and Go builds from whatever files a kept module cache holds without
shared by every build on the host, and Go builds from whatever files checking them again (only `go mod verify` does), so a file changed there
it holds without checking them again (only `go mod verify` does), so a would go unnoticed. Go's build cache also holds test results, which is why
file changed there would go unnoticed. both `go test` runs carry `-count=1`.
- Go's build cache (`/root/.cache/go-build`, where Go keeps it for
root), mounted on the steps that compile: `golangci-lint run`, both
`go test` runs and `go build`. Go reuses a compiled package only when
all its inputs are unchanged. The cache also holds test results, which
is why both `go test` runs carry `-count=1`.
- golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned
image keeps it), mounted on `golangci-lint run`. golangci-lint reuses
a package's findings only when the package's files and those of every
package it imports, `go.mod`, the linters and their settings in
`.golangci.yml`, and the golangci-lint version are all unchanged.
- If the project uses `//go:embed` directives that reference build artifacts - If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must (e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example: create placeholder files so the embed directives resolve. Example:
@@ -317,20 +292,12 @@ style conventions are in separate documents:
in the lint phase. The `golangci/golangci-lint` image is Debian-based and in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package (`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them. The test phase, lists in the same `RUN`, so the layer does not keep them:
on the Debian Go image, installs the libraries its tests need the same
way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so
Docker keeps it until the base image or the install line changes:
```dockerfile ```dockerfile
FROM golangci/golangci-lint@sha256:... AS lint
RUN apt-get update \ RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \ && apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
``` ```
- `.dockerignore` lets `.git` into the build context. It keeps out every git - `.dockerignore` lets `.git` into the build context. It keeps out every git
+4 -42
View File
@@ -29,40 +29,6 @@
# Completed Steps # Completed Steps
- re-vendor `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` from
`sneak/prompts` `next` at `0653579`: in the `lint`, `test` and `builder`
stages of the `Dockerfile` the module download, and in `builder` the
system-package install, now come before `ARG CACHEBUST`, so Docker keeps them
until the base image, the install line, `go.mod` or `go.sum` changes, and
every step that lints, tests or compiles still comes after it;
`golangci-lint run` also keeps golangci-lint's cache. README says the same
(2026-10-18, https://git.eeqj.de/sneak/sfdupes/issues/109)
- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so
`go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has
passed the build; README "Install" says so, says what `@main` and `v0.0.1`
give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the
early build without a database. When the linker set no version,
`sfdupes --version` prints the main module's version that Go recorded in the
binary, the tag or the pseudo-version of a `go install` build, and `dev` only
when neither is known (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/116)
- `scan` creates the database's directory where the database path leads: the
path with its last element cut off as text, never cleaned, so a `..` after a
symlinked directory is followed through the symlink, as when the file is
opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
symlink to `/data/x`, it creates `/data/new` and the database in it, where it
used to create `/srv/new` and then fail (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/119)
- `scan` never records its own database: once the database and its `.lock` file
are open, it notes the device and inode of the database file, its `-wal` and
`-shm` files and its `.lock` file, and the walk passes over a file with one of
them wherever the operands reach it, through a symlinked directory included,
printing no line and counting nothing. A record an earlier scan left for one
of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
- `scan` never records or reads anything that is not on a persistent filesystem - `scan` never records or reads anything that is not on a persistent filesystem
(`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types), (`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
matched by the device ID of each such filesystem, listed as the scan starts matched by the device ID of each such filesystem, listed as the scan starts
@@ -163,11 +129,7 @@
and the build stage installs `git` and `make` itself; a new push cancels the and the build stage installs `git` and `make` itself; a new push cancels the
workflow's older run on the same branch, and a run stops after 20 minutes; workflow's older run on the same branch, and a run stops after 20 minutes;
`.claude/settings.json` is deleted (2026-10-15, `.claude/settings.json` is deleted (2026-10-15,
https://git.eeqj.de/sneak/sfdupes/issues/95). Since https://git.eeqj.de/sneak/sfdupes/issues/95)
https://git.eeqj.de/sneak/sfdupes/issues/109 the module download in `lint`,
`test` and the build stage, and the install of `git` and `make` in the build
stage, come before `ARG CACHEBUST`, and Docker keeps them between builds;
every step that lints, tests or compiles still runs on every build
- with `-x`, an operand under another operand is walked on its own instead of - with `-x`, an operand under another operand is walked on its own instead of
being dropped when a directory from it (from its directory, for a regular being dropped when a directory from it (from its directory, for a regular
@@ -368,9 +330,9 @@
a `CHECK_EPOCH` build argument; since a `CHECK_EPOCH` build argument; since
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/` https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile` passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile`
declares before every step that lints, tests or compiles. Run as root, the declares first. Run as root, the tests fail
tests fail `TestScanHardlinkRunFailsTogether`, because root reads through the `TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)`
`chmod(0)` the test relies on, so the `test` phase runs them as `nobody` the test relies on, so the `test` phase runs them as `nobody`
- check the installed golangci-lint version in `script/bootstrap` instead of - check the installed golangci-lint version in `script/bootstrap` instead of
only its presence (2026-08-09, branch `bootstrap-version-check`, closes only its presence (2026-08-09, branch `bootstrap-version-check`, closes
https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
+8 -60
View File
@@ -109,8 +109,10 @@ func databasePath() (string, string, error) {
} }
// The working directory as the kernel reports it, with no symlink // The working directory as the kernel reports it, with no symlink
// in it, as README "Database" says. os.Getwd may return PWD // in it. os.Getwd may return PWD instead, which can name it through
// instead, which can name it through a symlink. // a symlink; a ".." after that symlink then leads somewhere else
// once the path is cleaned, as scan cleans it to create the
// database's directory.
wd, err := unix.Getwd() wd, err := unix.Getwd()
if err != nil { if err != nil {
return "", "", fmt.Errorf("resolve database path %s: %w", path, err) return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
@@ -123,24 +125,6 @@ func databasePath() (string, string, error) {
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
} }
// createDatabaseDirectory creates the directory the database at path
// goes in, and any directory above it, if missing. path is absolute, as
// databasePath returns it. The directory is path with its last element
// cut off as text, by filepath.Split. filepath.Dir would clean it,
// dropping each ".." together with the directory name before it; when
// that name is a symlink, the cleaned path names another directory than
// the one the system reaches when it opens path.
func createDatabaseDirectory(path string) error {
dir, _ := filepath.Split(path)
err := os.MkdirAll(dir, dbDirPerm)
if err != nil {
return fmt.Errorf("create database directory: %w", err)
}
return nil
}
// scanParams are the connection parameters for scan: read-write, with // scanParams are the connection parameters for scan: read-write, with
// WAL journaling and a busy timeout, so a report can run while a cron // WAL journaling and a busy timeout, so a report can run while a cron
// scan is in progress. closeScanDatabase leaves WAL mode again. // scan is in progress. closeScanDatabase leaves WAL mode again.
@@ -195,9 +179,9 @@ func openDB(path, params string) (*sql.DB, error) {
// that deleted it would let the next scan lock a new file while // that deleted it would let the next scan lock a new file while
// another still holds the old one. // another still holds the old one.
func lockScanDatabase(path string) (*os.File, error) { func lockScanDatabase(path string) (*os.File, error) {
err := createDatabaseDirectory(path) err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("create database directory: %w", err)
} }
// Like SQLite, this follows a symlink at path and creates its // Like SQLite, this follows a symlink at path and creates its
@@ -241,9 +225,9 @@ func lockScanDatabase(path string) (*os.File, error) {
// openScanDatabase opens the database for the scan subcommand, creating // openScanDatabase opens the database for the scan subcommand, creating
// the file, its parent directory, and the schema as needed. // the file, its parent directory, and the schema as needed.
func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) { func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
err := createDatabaseDirectory(path) err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("create database directory: %w", err)
} }
db, err := openDB(path, scanParams) db, err := openDB(path, scanParams)
@@ -261,42 +245,6 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
return db, nil return db, nil
} }
// databaseFiles returns, by device and inode, the files of the database
// at path that the scan keeps open: the database file, its -wal and
// -shm files and its .lock file, each named from the database's path
// with symlinks resolved, as SQLite and lockScanDatabase name them. The
// scan calls it once the database and the lock file are open, so that
// its walk passes over them however it reaches them, through a
// symlinked directory included. A file that does not exist then is left
// out, since no walk can reach it. There is no -journal file while the
// database is in WAL mode.
func databaseFiles(path string) ([]fileRec, error) {
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return nil, fmt.Errorf("resolve database path: %w", err)
}
var files []fileRec
for _, p := range []string{
resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
} {
fi, err := os.Lstat(p)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("database: %w", err)
}
dev, ino := inodeOfInfo(fi)
files = append(files, fileRec{path: p, dev: dev, ino: ino})
}
return files, nil
}
// closeScanDatabase switches the database at path from WAL back to // closeScanDatabase switches the database at path from WAL back to
// rollback-journal mode and closes it. Out of WAL mode the database // rollback-journal mode and closes it. Out of WAL mode the database
// file alone holds the whole database, so a reader needs no -wal or // file alone holds the whole database, so a reader needs no -wal or
-111
View File
@@ -5,7 +5,6 @@ import (
"database/sql" "database/sql"
"errors" "errors"
"fmt" "fmt"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"slices" "slices"
@@ -119,116 +118,6 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
} }
} }
// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
// through a symlink to a directory, then "..", to a missing directory.
// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
// places scan creates the database's directory, must each create the
// database there, and nothing at srv/new, where the path leads once
// cleaned as text.
func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
t.Parallel()
creates := map[string]func(ctx context.Context, path string) error{
"lockScanDatabase": func(_ context.Context, path string) error {
lock, err := lockScanDatabase(path)
if err == nil {
_ = lock.Close()
}
return err
},
"openScanDatabase": func(ctx context.Context, path string) error {
db, err := openScanDatabase(ctx, path)
if err == nil {
_ = db.Close()
}
return err
},
}
for name, create := range creates {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Symlink(filepath.Join(dir, "data", "x"),
filepath.Join(dir, "srv", "link"))
if err != nil {
t.Fatal(err)
}
err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
if err != nil {
t.Fatalf("%s: %v", name, err)
}
_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
if err != nil {
t.Errorf("database the path leads to: %v", err)
}
_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("srv/new: stat error %v, want it missing", err)
}
})
}
}
// TestDatabaseFiles opens a database through a symlink to its file:
// databaseFiles finds the database file and its -wal and -shm files
// beside the symlink's target, where SQLite keeps them, and leaves out
// the .lock file, which only a scan creates.
func TestDatabaseFiles(t *testing.T) {
t.Parallel()
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
db, err := openScanDatabase(t.Context(), link)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
files, err := databaseFiles(link)
if err != nil {
t.Fatal(err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.path)
}
want := []string{path, path + "-wal", path + "-shm"}
if !slices.Equal(got, want) {
t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
}
}
func TestOpenDatabaseUnversionedForeign(t *testing.T) { func TestOpenDatabaseUnversionedForeign(t *testing.T) {
t.Parallel() t.Parallel()
+1 -4
View File
@@ -43,17 +43,14 @@ const typesPerHelpLine = 6
// readMountInfo reads. // readMountInfo reads.
var errMountInfoLine = errors.New("unexpected line") var errMountInfoLine = errors.New("unexpected line")
// walkRules holds the rules for the walk that a scan sets up as it // walkRules holds the two rules for the walk that a scan sets up as it
// starts. excludes are the default excludes, or none with // starts. excludes are the default excludes, or none with
// --no-default-excludes. nonPersistent maps the device ID of every // --no-default-excludes. nonPersistent maps the device ID of every
// mounted filesystem that is not persistent to its type; the scan // mounted filesystem that is not persistent to its type; the scan
// records and reads nothing on one of those, whatever the flags. // records and reads nothing on one of those, whatever the flags.
// databaseFiles are the scan's own database files, which the walk
// passes over (see databaseFiles in db.go).
type walkRules struct { type walkRules struct {
excludes []string excludes []string
nonPersistent map[uint64]string nonPersistent map[uint64]string
databaseFiles []fileRec
} }
// defaultExcludes returns the default excludes of the system goos // defaultExcludes returns the default excludes of the system goos
-2
View File
@@ -24,5 +24,3 @@ require (
modernc.org/mathutil v1.7.1 // indirect modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect modernc.org/memory v1.11.0 // indirect
) )
retract v0.0.1 // early build without a database; does not work as the README describes
+3 -29
View File
@@ -28,7 +28,6 @@ import (
"io" "io"
"os" "os"
"runtime" "runtime"
"runtime/debug"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@@ -59,32 +58,10 @@ var errNoSubcommand = errors.New("no subcommand")
var errWorkersBelowOne = errors.New("--workers must be at least 1") var errWorkersBelowOne = errors.New("--workers must be at least 1")
// Version is the build version, injected at link time via -ldflags // Version is the build version, injected at link time via -ldflags
// (see the Makefile); empty when the linker set none, as for a binary // (see the Makefile); "dev" for a plain go build.
// built by go install. buildVersion turns it into what --version prints.
// //
//nolint:gochecknoglobals // written only by the linker //nolint:gochecknoglobals // written only by the linker
var Version string var Version = "dev"
// buildVersion returns the version that sfdupes --version prints
// (README "Install"): the version the linker set, when it set one;
// otherwise the main module's version that Go recorded in the binary,
// a tag such as v0.0.2 or the pseudo-version of an @main build; and
// "dev" when neither is known. info is what debug.ReadBuildInfo
// returns, nil when the binary carries none. Go records "(devel)" for
// a build that has no version, such as one inside a copy of the source
// without git.
func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
if linkerVersion != "" {
return linkerVersion
}
if info == nil || info.Main.Version == "" ||
info.Main.Version == "(devel)" {
return "dev"
}
return info.Main.Version
}
func main() { func main() {
// Once the reader of a stdout pipe has gone, as in "sfdupes report | // Once the reader of a stdout pipe has gone, as in "sfdupes report |
@@ -139,10 +116,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
var showVersion bool var showVersion bool
printVersion := runE(func(context.Context, []string) error { printVersion := runE(func(context.Context, []string) error {
info, _ := debug.ReadBuildInfo() _, err := fmt.Fprintf(stdout, "sfdupes %s\n", Version)
_, err := fmt.Fprintf(stdout, "sfdupes %s\n",
buildVersion(Version, info))
if err != nil { if err != nil {
return fmt.Errorf("write stdout: %w", err) return fmt.Errorf("write stdout: %w", err)
} }
+1 -38
View File
@@ -9,7 +9,6 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"runtime/debug"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -715,8 +714,6 @@ func TestRunVersion(t *testing.T) {
// README §Subcommands: the version is one line on stdout, with // README §Subcommands: the version is one line on stdout, with
// nothing on stderr, and exits 0. // nothing on stderr, and exits 0.
info, _ := debug.ReadBuildInfo()
for _, arg := range []string{"--version", "-v"} { for _, arg := range []string{"--version", "-v"} {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -725,7 +722,7 @@ func TestRunVersion(t *testing.T) {
t.Errorf("run(%s) = %d, want %d", arg, code, exitOK) t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
} }
want := "sfdupes " + buildVersion(Version, info) + "\n" want := "sfdupes " + Version + "\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("run(%s) stdout = %q, want %q", arg, got, want) t.Errorf("run(%s) stdout = %q, want %q", arg, got, want)
} }
@@ -736,40 +733,6 @@ func TestRunVersion(t *testing.T) {
} }
} }
func TestBuildVersion(t *testing.T) {
t.Parallel()
// README "Install": the linker's version wins; otherwise the main
// module's version that Go recorded; "dev" when neither is known.
tagged := &debug.BuildInfo{Main: debug.Module{Version: "v0.0.2"}}
pseudo := &debug.BuildInfo{Main: debug.Module{
Version: "v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1",
}}
devel := &debug.BuildInfo{Main: debug.Module{Version: "(devel)"}}
cases := []struct {
name string
linker string
info *debug.BuildInfo
want string
}{
{"linker set", "v0.0.2-3-g4e4a94d", tagged, "v0.0.2-3-g4e4a94d"},
{"linker set, no build info", "4e4a94d", nil, "4e4a94d"},
{"build info with a tag", "", tagged, "v0.0.2"},
{"build info with a pseudo-version", "", pseudo,
"v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1"},
{"(devel)", "", devel, "dev"},
{"no build info", "", nil, "dev"},
}
for _, c := range cases {
if got := buildVersion(c.linker, c.info); got != c.want {
t.Errorf("%s: buildVersion(%q, ...) = %q, want %q",
c.name, c.linker, got, c.want)
}
}
}
func TestRunVersionWriteFailureIsFatal(t *testing.T) { func TestRunVersionWriteFailureIsFatal(t *testing.T) {
t.Parallel() t.Parallel()
+11 -34
View File
@@ -118,8 +118,7 @@ func scanPhase(name string) string {
// rejects anything less. Before the lock is taken, the scan lists the // rejects anything less. Before the lock is taken, the scan lists the
// filesystems that are not persistent, on which the walk records // filesystems that are not persistent, on which the walk records
// nothing; unless noDefaultExcludes is set, the walk does not enter the // nothing; unless noDefaultExcludes is set, the walk does not enter the
// default excludes either. Once the database is open, the scan notes // default excludes either.
// its files, which the walk passes over.
func runScan(ctx context.Context, roots []string, workers int, func runScan(ctx context.Context, roots []string, workers int,
oneFS, noDefaultExcludes bool, oneFS, noDefaultExcludes bool,
) error { ) error {
@@ -174,11 +173,6 @@ func runScan(ctx context.Context, roots []string, workers int,
defer closeScanDatabase(ctx, db, dbPath) defer closeScanDatabase(ctx, db, dbPath)
rules.databaseFiles, err = databaseFiles(dbPath)
if err != nil {
return err
}
st, err := syncScan(ctx, db, roots, workers, oneFS, rules) st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
if errors.Is(err, context.Canceled) { if errors.Is(err, context.Canceled) {
return interrupted(st.walked) return interrupted(st.walked)
@@ -716,15 +710,6 @@ func sameInode(a, b fileRec) bool {
return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
} }
// isDatabaseFile reports whether f, a file the walk reached, is one of
// the scan's own database files in rules, which the walk passes over
// without a line and without counting it.
func isDatabaseFile(f fileRec, rules walkRules) bool {
return slices.ContainsFunc(rules.databaseFiles, func(d fileRec) bool {
return sameInode(d, f)
})
}
// hashPhase hashes every queued file with hashSignature — the head and // hashPhase hashes every queued file with hashSignature — the head and
// tail of a file of headTailMin or more, the whole file below that — // tail of a file of headTailMin or more, the whole file below that —
// committing completed records to the database in batches as results // committing completed records to the database in batches as results
@@ -1158,7 +1143,7 @@ func startWalk(ctx context.Context, roots []string, oneFS bool,
go func() { go func() {
initial := make([]dirJob, 0, len(roots)) initial := make([]dirJob, 0, len(roots))
for _, root := range roots { for _, root := range roots {
initial = append(initial, seedRoot(ctx, root, rules, events)...) initial = append(initial, seedRoot(ctx, root, events)...)
} }
dispatchDirs(ctx, initial, jobs, subdirs) dispatchDirs(ctx, initial, jobs, subdirs)
@@ -1212,14 +1197,13 @@ func operandWarning(root string, fi fs.FileInfo) string {
} }
// seedRoot turns one PATH operand into the walk's starting state: a // seedRoot turns one PATH operand into the walk's starting state: a
// regular-file operand is statted and emitted directly, unless it is // regular-file operand is statted and emitted directly, and a directory
// one of the scan's own database files in rules, and a directory
// operand becomes an initial job. walkableRoots has already dropped // operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since // every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records // is warned about and skipped here; it is still a root, so the records
// stored beneath it, other than those under another root, are deleted // stored beneath it, other than those under another root, are deleted
// as unverified. // as unverified.
func seedRoot(ctx context.Context, root string, rules walkRules, func seedRoot(ctx context.Context, root string,
events chan<- walkEvent, events chan<- walkEvent,
) []dirJob { ) []dirJob {
fi, err := os.Lstat(root) fi, err := os.Lstat(root)
@@ -1246,17 +1230,14 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
} }
dev, ino := inodeOfInfo(fi) dev, ino := inodeOfInfo(fi)
rec := fileRec{
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: root, path: root,
size: fi.Size(), size: fi.Size(),
mtime: fi.ModTime(), mtime: fi.ModTime(),
dev: dev, dev: dev,
ino: ino, ino: ino,
} }})
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
return nil return nil
} }
@@ -1391,8 +1372,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
// regular file) between the directory read and the lstat is warned // regular file) between the directory read and the lstat is warned
// about and skipped. A file on a filesystem in rules that is not // about and skipped. A file on a filesystem in rules that is not
// persistent, as through a bind mount of one file, is not recorded: it // persistent, as through a bind mount of one file, is not recorded: it
// gets an event naming it instead. One of the scan's own database files // gets an event naming it instead.
// in rules gets no event at all.
func emitFile(ctx context.Context, p string, e fs.DirEntry, func emitFile(ctx context.Context, p string, e fs.DirEntry,
rules walkRules, events chan<- walkEvent, rules walkRules, events chan<- walkEvent,
) { ) {
@@ -1418,17 +1398,14 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
} }
dev, ino := inodeOfInfo(info) dev, ino := inodeOfInfo(info)
rec := fileRec{
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: p, path: p,
size: info.Size(), size: info.Size(),
mtime: info.ModTime(), mtime: info.ModTime(),
dev: dev, dev: dev,
ino: ino, ino: ino,
} }})
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
} }
// subdirJob applies the descent rules to directory p: never enter // subdirJob applies the descent rules to directory p: never enter
+1 -97
View File
@@ -1171,7 +1171,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
var initial []dirJob var initial []dirJob
for _, root := range roots { for _, root := range roots {
for _, job := range seedRoot(ctx, root, walkRules{}, events) { for _, job := range seedRoot(ctx, root, events) {
job.rootDev = devices[root] job.rootDev = devices[root]
initial = append(initial, job) initial = append(initial, job)
} }
@@ -2501,99 +2501,3 @@ func TestUnderRoot(t *testing.T) {
} }
} }
} }
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
// database, reached directly and through a symlinked directory. The
// walk passes over the database file, its -wal and -shm files and its
// .lock file with no line and no count, and the record an earlier scan
// left for the .lock file is deleted. A rescan of the unchanged tree
// updates nothing, and report lists none of the database's files.
func TestRunScanPassesOverDatabase(t *testing.T) {
for _, viaSymlink := range []bool{false, true} {
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
base := t.TempDir()
tree := filepath.Join(base, "real", "tree")
operand := tree
writeFile(t, tree, "a", pattern(1, 100))
writeFile(t, tree, "b", pattern(1, 100))
writeFile(t, tree, "empty1", nil)
writeFile(t, tree, "empty2", nil)
if viaSymlink {
err := os.Symlink(filepath.Join(base, "real"),
filepath.Join(base, "link"))
if err != nil {
t.Fatal(err)
}
operand = filepath.Join(base, "link", "tree")
}
dbPath := filepath.Join(tree, "db.sqlite")
t.Setenv(databaseEnv, dbPath)
db, err := openScanDatabase(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
err = applyChanges(t.Context(), db, []scanRec{
{path: filepath.Join(operand, "db.sqlite.lock")},
}, nil, nil)
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, dbPath)
assertScanSummary(t,
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
operand)
assertScanSummary(t,
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
operand)
report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") ||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
t.Errorf("report = %q, want the empty files and none of "+
"the database's files", report)
}
})
}
}
// TestRunScanPassesOverDatabaseOperands scans the database file and its
// .lock file given as operands themselves: neither is recorded or
// counted.
func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
dbPath := testDBPath(t)
t.Setenv(databaseEnv, dbPath)
// The first scan creates both files.
scanOK(t, t.TempDir())
assertScanSummary(t,
"0 files seen (0 added, 0 updated, 0 removed, 0 unchanged)",
dbPath, dbPath+".lock")
}
// assertScanSummary scans operands and fails the test unless the
// summary line gives counts with nothing skipped, and the line naming
// the database is the only one that names one of its files.
func assertScanSummary(t *testing.T, counts string, operands ...string) {
t.Helper()
stderr := scanOK(t, operands...)
want := "scan: " + counts + ", 0 skipped\n"
if !strings.Contains(stderr, want) {
t.Errorf("scan %q: stderr = %q, want %q", operands, stderr, want)
}
if strings.Count(stderr, "db.sqlite") != 1 {
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
"database to name one of its files", operands, stderr)
}
}
+4 -5
View File
@@ -2,12 +2,11 @@
# script/cibuild: run the CI build. It bootstraps first: a CI runner # script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs # checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do. # the formatter on the host, which a pristine checkout cannot do.
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST # A new CACHEBUST value reruns every RUN step, since a cached gate phase
# line, since a cached gate phase is a check that did not run; unlike # is a check that did not run; unlike --no-cache, it keeps Go's build cache.
# --no-cache, it keeps the layers above that line and the cache mounts.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST # This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# before every step that lints, tests or compiles; a stage without that # first; a stage without that line is served from cache on an unchanged
# line is served from cache on an unchanged tree, with no warning. # tree, with no warning.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+4 -5
View File
@@ -1,12 +1,11 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname. # Identical in all repos; the tag comes from script/projectname.
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST # A new CACHEBUST value reruns every RUN step, since a cached gate phase
# line, since a cached gate phase is a check that did not run; unlike # is a check that did not run; unlike --no-cache, it keeps Go's build cache.
# --no-cache, it keeps the layers above that line and the cache mounts.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST # This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# before every step that lints, tests or compiles; a stage without that # first; a stage without that line is served from cache on an unchanged
# line is served from cache on an unchanged tree, with no warning. # tree, with no warning.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+5 -7
View File
@@ -5,13 +5,11 @@
# make its answer untrustworthy. # make its answer untrustworthy.
# #
# The phase is not the last stage in the file, so it is built only when # The phase is not the last stage in the file, so it is built only when
# --target names it. A new CACHEBUST value reruns every RUN step after # --target names it. A new CACHEBUST value reruns every RUN step, since
# the ARG CACHEBUST line, since a cached lint layer is a lint that did # a cached lint layer is a lint that did not run; unlike --no-cache, it
# not run; unlike --no-cache, it keeps the layers above that line and # keeps Go's build cache. This relies on every stage of the Dockerfile
# the cache mounts. This relies on every stage of the Dockerfile # declaring ARG CACHEBUST first; a stage without that line is served
# declaring ARG CACHEBUST before every step that lints, tests or # from cache on an unchanged tree, with no warning.
# compiles; a stage without that line is served from cache on an
# unchanged tree, with no warning.
# --output type=cacheonly writes no image, since nothing uses one. # --output type=cacheonly writes no image, since nothing uses one.
set -eu set -eu
+5 -6
View File
@@ -3,12 +3,11 @@
# and this builds that phase alone, on the same terms as script/lint: # and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when # --target because a phase that is not the last stage is built only when
# named, and a new CACHEBUST value because a cached test layer is a test # named, and a new CACHEBUST value because a cached test layer is a test
# that did not run; unlike --no-cache, it keeps the layers above the ARG # that did not run; unlike --no-cache, it keeps Go's build cache. This
# CACHEBUST line and the cache mounts. This relies on every stage of the # relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
# Dockerfile declaring ARG CACHEBUST before every step that lints, tests # a stage without that line is served from cache on an unchanged tree,
# or compiles; a stage without that line is served from cache on an # with no warning. --output type=cacheonly writes no image, since
# unchanged tree, with no warning. --output type=cacheonly writes no # nothing uses one.
# image, since nothing uses one.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"