Author SHA1 Message Date
clawbot c64bbbb78e Re-vendor the canonical files from sneak/prompts at c55a0cb (closes #95)
check / check (push) Waiting to run
Every vendored file, REPO_POLICIES.md and every model script is the
copy at sneak/prompts c55a0cb, with this repository's own entries kept
after the canonical content. Lint and test are phases of the Dockerfile
that write no image, built uncached. make test runs the suite under the
race detector as nobody, because root reads the files the tests make
unreadable. Dockerfile.lint, script/verify-lint-image-pin and
make test-race are gone. Prettier runs on the host, from the node and
yarn that script/bootstrap installs. golangci-lint v2.14.0 raises no
findings. .claude/settings.json is deleted.

Deviation: the set comes from c55a0cb on next rather than dd4027b, as
the instructions on sneak/prompts#78 allow.

Model: opus-5-5
2026-10-08 01:32:30 +00:00
clawbot 5900feb515 Store mtime to the nanosecond so a same-second rewrite is re-hashed (closes #12)
check / check (push) Waiting to run
scan recorded mtime in whole seconds, so a file rewritten in place at
the same size within the same second as its recorded mtime was classed
unchanged and kept its old hashes. The files table keeps mtime as whole
Unix seconds and gains mtime_nsec, the nanoseconds within that second.
scan holds the mtime as a time.Time and decides "newer" by comparing
Unix() and then Nanosecond(), so any time a filesystem can record
compares in the right order; After would misorder one too late for a
time.Time to hold without wrapping. The walk, a file given as an
operand, and the content phase's recheck all move over. PRAGMA
user_version stays 1, per the owner's ruling. README states what both
columns hold.

Model: opus-5-5
2026-10-08 02:51:53 +02:00
25 changed files with 674 additions and 320 deletions
+3
View File
@@ -13,3 +13,6 @@ indent_style = tab
[*.go]
indent_style = tab
# This repository's own sections, such as one for another language it
# uses, go below this comment, and a re-vendor keeps them.
+9
View File
@@ -1,11 +1,20 @@
name: check
on: [push]
# Free the shared runner: a new push cancels only the same branch's older run.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
check:
runs-on: ubuntu-latest
# Free the shared runner from a hung build.
timeout-minutes: 20
steps:
# actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
# script/cibuild needs no token, so none is left in .git/config.
with:
persist-credentials: false
# All history and tags, so git describe finds the version tag.
fetch-depth: 0
- run: script/cibuild
+5 -3
View File
@@ -27,7 +27,7 @@ node_modules/
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
# its own negation at the end of this file, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
@@ -46,11 +46,13 @@ node_modules/
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, kept after the canonical content above.
# This repository's own entries, such as its build outputs, go below
# this comment, and a re-vendor keeps them. Anchor a binary built at the
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
/sfdupes
*.log
*.out
*.test
/sfdupes
# A scan database lists every path it scanned.
*.sqlite
+2
View File
@@ -25,6 +25,8 @@ linters:
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
# Misses findings at random in v2.14.0; back once a pinned release fixes it
- canonicalheader
settings:
lll:
line-length: 88
+12 -36
View File
@@ -1,6 +1,6 @@
# Lint phase, built alone by script/lint. The tools are invoked directly
# rather than through `make lint` or `make fmt-check`, which run docker
# themselves and so cannot run inside a build step.
# rather than through `make lint`, which runs docker itself and so cannot
# run inside a build step.
# golangci/golangci-lint:v2.14.0, 2026-10-07
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src
@@ -8,9 +8,9 @@ COPY go.mod go.sum ./
RUN go mod download
COPY . .
# The gofmt half of `make fmt-check`; the markdown stage is its prettier
# half. gofmt's output is assigned to a variable first so that its own
# exit status, as when it cannot parse a file, still fails the step.
# The gofmt half of `make fmt-check`. gofmt's output is assigned to a
# variable first so that its own exit status, as when it cannot parse a
# file, still fails the step.
RUN files="$(gofmt -s -l .)" && \
if [ -n "$files" ]; then \
echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \
@@ -38,43 +38,19 @@ RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
# Prettier stage: the prettier that formats this repository's Markdown,
# never installed on a host. script/fmt and script/fmt-check build this
# stage alone and run it with the repository mounted on /src. prettier
# is installed in /tools so that the repository, mounted or copied onto
# /src, cannot hide it.
# node:22-alpine, 2026-02-22
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS prettier
WORKDIR /tools
# yarn.lock pins prettier by hash, and --frozen-lockfile fails rather
# than install anything yarn.lock does not name.
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
ENV PATH=/tools/node_modules/.bin:$PATH
WORKDIR /src
# Markdown stage: the Markdown half of `make fmt-check`, as a gate.
FROM prettier AS markdown
COPY . .
RUN prettier --check '**/*.md' --tab-width 4 --prose-wrap always
# Build stage. Nothing is wanted from the lint, test and markdown stages;
# the copies are what make BuildKit build them first, so this stage
# cannot run unless all three passed.
# Build stage. Nothing is wanted from either phase above; the copies are
# what make BuildKit build them first, so this stage cannot run unless
# lint and test passed.
# golang:1.25-alpine, 2026-07-23
FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
COPY --from=markdown /src/go.sum /dev/null
WORKDIR /src
# script/bootstrap installs the git and make this image lacks, and ends
# in `go mod download`.
COPY script/ script/
COPY go.mod go.sum ./
RUN script/bootstrap
RUN apk add --no-cache git make
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# The version stamped into the binary: the VERSION build argument when
+51 -48
View File
@@ -314,7 +314,8 @@ All three subcommands operate on a single SQLite database file:
CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- Unix seconds, from lstat
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples
@@ -323,18 +324,22 @@ All three subcommands operate on a single SQLite database file:
```
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
UTF-8. `mtime` is used only for change detection; it is not part of the
duplicate key. For a file under 10 MiB `head`, `tail`, and `content` all
hold the whole-file hash (that range is hashed in full, with no end
windows); for a larger file `head` and `tail` hold the first- and last-64
KiB hashes and `content` the whole-file or sampled hash. All three are empty
strings when the file has never been hashed because its size was unique as
of the last scan that covered it. For a file of 10 MiB or more, `content`
stays empty until the content phase of a scan (see "`scan` mode" below) has
read the file. A record with an empty `content` is never part of a duplicate
group, though it still defines the file for tree reconstruction. The
`files_signature` index lets SQLite group the records by signature for
`report` without sorting the whole table.
UTF-8. `mtime` and `mtime_nsec` hold the file's mtime to the nanosecond:
`mtime` the whole Unix seconds, rounded down, and `mtime_nsec` the
nanoseconds past that second. Split this way they hold any mtime a
filesystem can record, one before 1678 or after 2262 included, which a
single 64-bit count of nanoseconds cannot. They are used only for change
detection and are not part of the duplicate key. For a file under 10 MiB
`head`, `tail`, and `content` all hold the whole-file hash (that range is
hashed in full, with no end windows); for a larger file `head` and `tail`
hold the first- and last-64 KiB hashes and `content` the whole-file or
sampled hash. All three are empty strings when the file has never been
hashed because its size was unique as of the last scan that covered it. For
a file of 10 MiB or more, `content` stays empty until the content phase of a
scan (see "`scan` mode" below) has read the file. A record with an empty
`content` is never part of a duplicate group, though it still defines the
file for tree reconstruction. The `files_signature` index lets SQLite group
the records by signature for `report` without sorting the whole table.
### Duplicate detection
@@ -421,6 +426,9 @@ operands:
once its size, `head`, and `tail` match another record's.
- A file whose mtime is newer than recorded, or whose size differs, is processed
as if new: re-hashed, or recorded without hashes, per the shared-size rule.
Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the
filesystem gives it a later mtime than recorded, even within the same second.
- A database record whose path lies under one of the scanned operands but was
not successfully processed this run is deleted. This removes records for
deleted files. It also removes records for paths that failed to stat or hash
@@ -738,50 +746,48 @@ entrypoints are:
- `script/bootstrap` — install everything needed to build and develop this
repository, idempotently, assuming nothing is present. `git`, `make`, and `go`
come from the first of nix, apt, brew, or apk found on the host, and are
presence-checked only. `golangci-lint` and prettier are deliberately **not**
installed: they run in Docker (see `script/lint` and `script/fmt`) and never
from a host install, so there is no host copy to drift from the pin. A missing
`docker` is warned about rather than installed or treated as fatal:
`make build` works without it, and testing, linting, formatting and the image
build need it. Ends with `go mod download`.
presence-checked only. An installed node is used as it is; otherwise node
22.17.0 is installed through nvm, which comes from a release archive whose
sha256 the script checks. A yarn already on `PATH` is used as it is; otherwise
yarn 1.22.22 is activated through corepack, or installed with `npm` when there
is no corepack. `yarn install --frozen-lockfile` then installs the prettier
that `package.json` and `yarn.lock` pin. `golangci-lint` is never installed:
it runs in Docker (see `script/lint`). `docker` is not installed either;
testing, linting and the image build need it. Ends with `go mod download`.
- `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`.
- `script/projectname` — print this project's name (`sfdupes`). Scripts that
need the name call it, so they stay identical across repositories.
- `script/test` — run the test suite under the race detector, with a 90-second
timeout and coverage enabled, rerunning verbosely on failure so the logs show
which test failed. It builds the `Dockerfile`'s `test` phase alone, tagged
`sfdupes-test`. The race detector needs cgo and a C compiler, which the build
never uses, so the phase starts from a digest-pinned Debian `golang` image,
which has `gcc`. The tests run as `nobody`, because several of them make a
file unreadable and root reads it anyway.
which test failed. It builds the `Dockerfile`'s `test` phase alone and writes
no image. The race detector needs cgo and a C compiler, which the build never
uses, so the phase starts from a digest-pinned Debian `golang` image, which
has `gcc`. The tests run as `nobody`, because several of them make a file
unreadable and root reads it anyway.
- `script/lint` — run the linter. It builds the `Dockerfile`'s `lint` phase
alone, tagged `sfdupes-lint`: in the digest-pinned `golangci/golangci-lint`
alone and writes no image: in the digest-pinned `golangci/golangci-lint`
image, the gofmt check, `golangci-lint config verify` and `golangci-lint run`
run as build steps, so a successful build is a clean lint. The linter is never
run on the host, which makes a working `docker` the one prerequisite for
linting.
- `script/fmt` — format in place: the Go sources with `gofmt -s -w`, and every
Markdown file with prettier, at the settings in `.prettierrc` (4-space
indents, prose wrapped at 80 columns). prettier is pinned by hash through
`package.json` and `yarn.lock` and never installed on the host: this builds
the `Dockerfile`'s `prettier` stage, a digest-pinned node image into which
`yarn install --frozen-lockfile` installs it, tagged `sfdupes-prettier`, and
runs that with the repository mounted, as the calling user. Needs `docker`,
and because of the mount, unlike `script/lint`, a local docker daemon.
- `script/fmt-check` — the read-only counterpart of `script/fmt`, with the
repository mounted read-only: prints any unformatted file and exits non-zero
instead of writing. gofmt and prettier both run every time, and each names
itself when it fails. The `Dockerfile` runs the same two checks as gates: the
gofmt check in its `lint` phase, prettier in its `markdown` stage.
indents, prose wrapped at 80 columns). Both run on the host, prettier through
yarn at the version `yarn.lock` pins. When yarn is not on `PATH`, this loads
the node 22.17.0 that `script/bootstrap` installed through nvm.
- `script/fmt-check` — the read-only counterpart of `script/fmt`: prints any
unformatted file and exits non-zero instead of writing. gofmt and prettier
both run every time, and each names itself when it fails. The `Dockerfile`'s
`lint` phase runs the same gofmt check.
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in
that order. Modifies nothing. Needs `docker`, because all three do.
that order. Modifies nothing. The first two need `docker`, the third what
`script/bootstrap` installs.
- `script/docker` — build the Docker image, tagged with the name from
`script/projectname`, passing the output of
`git describe --tags --always --dirty` (or `unknown` when that is empty) as
the `VERSION` build argument. The `Dockerfile`'s build stage depends on its
`lint`, `test` and `markdown` stages, so this is also the check a developer or
reviewer runs by hand.
`lint` and `test` phases, so this also runs the tests and the linter.
- `script/cibuild` — run `script/bootstrap`, then `script/check`, then build the
image as `script/docker` does. This is what the Gitea workflow runs on push; a
successful run means every check passed. The tests and the linter run twice:
@@ -789,16 +795,13 @@ entrypoints are:
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a
no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then
`script/check`.
- `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit`. The hook is written to the common git directory, so the
main checkout and every worktree share it.
- `script/install-precommit` — install the git pre-commit hook, as
`.git/hooks/pre-commit`, that runs `script/precommit`.
Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree
Docker would serve the gate steps from its cache, and the build would pass
having run no test and no linter. Every run therefore downloads the Go modules
again and needs the network. `script/test` and `script/lint` send the daemon
only a build context, so they work against a remote docker daemon; `script/fmt`
and `script/fmt-check` mount the repository and need a local one.
again and needs the network.
## Build
@@ -815,10 +818,10 @@ compile recipe:
- `make lint` — run `golangci-lint` with the repo config and the gofmt check, in
Docker (see `script/lint`); requires `docker`.
- `make fmt` / `make fmt-check` — format the Go sources and the Markdown /
verify formatting without writing; requires `docker`, for prettier (see
`script/fmt`).
verify formatting without writing, on the host; requires `go` and what
`make bootstrap` installs (see `script/fmt`).
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires
`docker`.
`docker` and what `make bootstrap` installs.
- `make docker` — build the Docker image, which runs the gates as build stages.
- `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary.
+58 -25
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-10-04
last_modified: 2026-10-07
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -118,8 +118,8 @@ style conventions are in separate documents:
and nothing else:
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
docker build --no-cache --target lint --output type=cacheonly .
docker build --no-cache --target test --output type=cacheonly .
```
**A stage that is not the last one in the file is built only when the final
@@ -129,10 +129,15 @@ style conventions are in separate documents:
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
**The gate builds write no image.** With `--output type=cacheonly` the phase
runs and a failing step fails the build, but the result is not exported.
Nothing uses those images, and writing one out is slow: a Go test phase's
image holds the toolchain and every compiled package. A build given neither
`--output` nor `-t` writes an untagged image and leaves it dangling, on
every developer host and every CI runner. `script/cibuild` and
`script/docker` build the image that ships and tag it, so each build
replaces the previous image; each assigns the tag on its own line before the
build, so `set -e` stops it where `script/projectname` fails.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
@@ -278,20 +283,39 @@ style conventions are in separate documents:
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
A checkout whose `.git` is a file (a linked worktree, or a repository
checked out as a submodule) is the exception: that file points to a git
directory outside the build context, so the build cannot read the version
and a plain `docker build .` fails; pass the version with
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
do.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
runs `script/cibuild` on push, and checks out the repo as its only other step,
with `persist-credentials: false`: `script/cibuild` needs no token, and
without it the checkout leaves the job's token in `.git/config` for every
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
tags `git describe` needs: by default it clones shallow with no tags, and a
tagged repository's CI build would stamp a bare short commit id. The
workflow's `concurrency` block groups runs by workflow and branch
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
so a new push cancels the older run on the same branch, queued or running, and
no other: runs for replaced commits do not hold up the shared runner.
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
so a successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
from a run of its own gates rather than from a cache entry. The `check` job
sets `timeout-minutes: 20`, so a hung build frees the shared runner after 20
minutes. That allows for the three Docker builds described above (the test
phase, the lint phase, then the image), each held to the 5-minute Docker build
limit below, plus the bootstrap. A separate workflow limited to `main` by a
`branches` list under `on: push` cannot be checked by review: to try a change
to it, add the feature branch to that list and push, then remove the branch
from the list again before merging. Keep any job in it that publishes behind
`if: github.ref_name == 'main'`, so the run from the feature branch publishes
nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -384,9 +408,12 @@ style conventions are in separate documents:
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
`node_modules/`, and the repo's own build outputs. Fetch the standard
`.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
own entries, such as its binaries; a re-vendor replaces the standard part and
keeps those entries. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
@@ -434,13 +461,15 @@ style conventions are in separate documents:
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
# The version and the tag each get their own line: a failing command
# substitution inside an argument does not trip `set -e`, so the inline
# form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$(script/projectname)"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
-t "$tag" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
@@ -452,8 +481,8 @@ style conventions are in separate documents:
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
checkout action clones shallow and fetches no tags, so the canonical
`.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
@@ -636,7 +665,11 @@ style conventions are in separate documents:
Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation
settings.
settings: the standard file from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
one for another language it uses. A re-vendor replaces the standard part and
keeps those sections.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
+64 -82
View File
@@ -29,27 +29,43 @@
# Completed Steps
- re-vendor the canonical files from `sneak/prompts` commit `dd4027b`:
golangci-lint v2.14.0; lint and test are phases of the `Dockerfile`, and
`make test` runs the suite under the race detector, so `Dockerfile.lint`,
`script/verify-lint-image-pin` and `make test-race` are gone; every
`docker build` in `script/` passes `--no-cache`; `.claude/settings.json` is
deleted (2026-10-07, https://git.eeqj.de/sneak/sfdupes/issues/95)
- re-vendor the canonical files and model scripts from `sneak/prompts` `next` at
`c55a0cb`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile`
that write no image, and `make test` runs the suite under the race detector,
so `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race` are
gone; every `docker build` in `script/` passes `--no-cache`; prettier runs on
the host, from the node and yarn `script/bootstrap` installs, so the
`prettier` and `markdown` stages are gone and the build stage installs `git`
and `make` itself; a new push cancels the workflow's older run on the same
branch, and a run stops after 20 minutes; `.claude/settings.json` is deleted
(2026-10-08, https://git.eeqj.de/sneak/sfdupes/issues/95)
- `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus
`mtime_nsec`, and compares it at that resolution, so a same-size rewrite
within the same second is re-hashed (2026-10-07,
https://git.eeqj.de/sneak/sfdupes/issues/12)
- cut the narration from `TODO.md` Completed Steps and from the comments in
`script/` and both Dockerfiles; §Workflow now branches from and merges to
`next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
`script/`, `Dockerfile` and `Dockerfile.lint` (gone since
https://git.eeqj.de/sneak/sfdupes/issues/95); §Workflow now branches from and
merges to `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
- `make test-race` runs the test suite under the race detector in a cgo-enabled
- `make test-race` ran the test suite under the race detector in a cgo-enabled
container, outside `make check` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/18)
https://git.eeqj.de/sneak/sfdupes/issues/18). Since
https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
under the race detector, in the `Dockerfile`'s Debian-based `test` phase, and
`make test-race` is gone
- a bare `docker build .` fails with a message naming `script/cibuild` and
`script/docker` instead of serving the gates from cache (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/39)
- a bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
rather than serve the gates from cache (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/39). Since
https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
`REPO_POLICIES.md` requires, and may serve the gates from cache; the builds in
`script/` pass `--no-cache`, so theirs always run
- `make fmt` and `make fmt-check` run prettier over all Markdown, in Docker, and
CI checks it; all Markdown reformatted (2026-10-04,
- `make fmt` and `make fmt-check` run prettier over all Markdown, and CI checks
it; all Markdown reformatted (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/19)
- `script/lint` writes no image, so a run no longer leaves an untagged one
@@ -92,9 +108,12 @@
- README documents install, Docker, a daily cron scan and how to read and check
the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
- the `Dockerfile` build stage keeps the Go module cache out of `builder`'s home
and copies the sources with `--chown`, so no `chown -R` walks them
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43)
- the `Dockerfile` build stage kept the Go module cache out of `builder`'s home
and copied the sources with `--chown`, so no `chown -R` walked them
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43). Since
https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
nothing changes owner: the build stage only compiles, as root, and the tests
run as `nobody` in the `test` phase
- `--version` prints `sfdupes VERSION` to stdout; README documents it and
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
@@ -158,82 +177,45 @@
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` (2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/25): both pins are now the policy
https://git.eeqj.de/sneak/sfdupes/issues/25): both pins became the policy
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
without the false `(Debian-based)` note or the tag; digest unchanged.
`script/verify-lint-image-pin` still matches the tagless form, and a tag on
one side only is caught as a plain mismatch.
without the false `(Debian-based)` note or the tag. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
holds the only golangci-lint pin, in that form, so `Dockerfile.lint` and
`script/verify-lint-image-pin`, which compared the two pins, are gone.
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
owner ruling the linter is never installed on a host. `Dockerfile.lint` copies
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
runs `golangci-lint config verify` and `golangci-lint run` as build steps;
`script/lint` builds it. `script/bootstrap` no longer installs or pins the
linter, and warns rather than fails when `docker` is absent;
`ENV PATH=/home/builder/go/bin:$PATH` went with its `go install`.
`script/verify-linter-pin` is retired; `script/verify-lint-image-pin`, a gate
in both files, compares their two `FROM` lines and restates neither pin.
Traps: nothing inside an image build may shell out to docker, so the
`Dockerfile` lint stage calls `golangci-lint` directly and the build stage
runs `make test` and `make fmt-check` instead of `make check`, through `make`
because the Makefile's `export CGO_ENABLED = 0` only reaches what it invokes.
`COPY --from=lint /src/go.sum /dev/null` replaces the copied linter binary as
the only edge making the build stage wait for lint; dropping it would end
fail-fast linting under a still-green build. `golangci-lint config verify`,
included per the ruling, validates from an embedded schema with no network
call, but `go mod download` above the gates still needs the network on a cold
cache. Verified: `make lint` green with no `golangci-lint` on `PATH`; two
back-to-back `script/lint` runs on an untouched tree both ran the linter
(27.7s and 28.7s in the lint step, `COPY . .` `CACHED` above); a planted
unused variable failed `script/lint`, and failed `make docker` at `[lint 9/9]`
with the build stage stopped at `[builder 3/12]`; the drift guard fails on a
tag-only, a digest-only and an unreadable reference, naming both sides; under
`--network none` config verify passes a valid config and rejects an invalid
one; `make docker` green in 5m35s with all six gates run (lint 37.6s, test
25.2s reporting `ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not
`(cached)`); in the builder image with the Go test cache off, `--user 0:0`
still fails `TestScanHardlinkRunFailsTogether` where the unprivileged user
passes. Noted for follow-up, not fixed here: `golangci-lint` warns that
`gomodguard` is deprecated since v2.12.0 in favour of `gomodguard_v2`.
owner ruling the linter is never installed on a host, and
`golangci-lint config verify` runs before `golangci-lint run`.
`script/bootstrap` stopped installing or pinning the linter, and
`script/verify-linter-pin` was retired. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
`Dockerfile`'s `lint` phase, which `script/lint` builds alone and the build
stage depends on through `COPY --from=lint /src/go.sum /dev/null`;
`Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
an image build may run docker, so the phase calls `golangci-lint` directly.
- install the Docker build stage's prerequisites by running `script/bootstrap`
instead of `apk add --no-cache make` inline (2026-08-09, branch
`dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42):
the stage copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
which ends in `go mod download`, so the separate call to it is gone.
`COPY --from=lint /usr/bin/golangci-lint` stays and moves above the bootstrap
layer: it is the only edge making this stage depend on the lint stage, so
deleting it would end fail-fast linting silently. A new
`script/verify-linter-pin`, run in the build stage before bootstrap, fails the
build naming both versions unless that copied binary is the version
`script/bootstrap` pins; a pin it cannot read is a hard failure, not a skip.
`$GOPATH/bin` joins `PATH`, where bootstrap's `go install` lands. The `chown`
and `USER builder` still precede `make check`. Verified: the guard fails the
build with both versions named when the lint stage's linter is faked to
another version, and passes an unmodified build; bootstrap runs clean under
Alpine's `sh` and `apk`, finding the copied linter already at the pin; a
second build served the bootstrap and dependency layers `CACHED` while both
gates ran; a planted `unused` finding failed the build at the lint gate in
48.9s with the build stage's `make check` never starting; and the suite run in
the image as `--user 0:0` fails `TestScanHardlinkRunFailsTogether`, so the
drop to the unprivileged user is still needed. That last check needs the Go
test cache off: as root it first reported `ok ... (cached)`, reusing the
build-time result. Build times on a noisy shared host: 2m13s on an unchanged
tree, 2m17s and 4m29s after a source change, 5m14s cold, which breaches the
policy ceiling; `chown -R builder:builder /src /home/builder` walks the module
cache and alone varied from 77s to 210s across those builds, and `main`
measured 5m03s cold with a 209s `chown`. Filed as
https://git.eeqj.de/sneak/sfdupes/issues/43
`dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42),
with `script/verify-linter-pin` failing the build unless the linter copied
from the lint stage was the version `script/bootstrap` pinned. Builds then
took up to 5m14s cold, mostly in a `chown -R` of the module cache, filed as
https://git.eeqj.de/sneak/sfdupes/issues/43. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
`make` with `apk add --no-cache` and only compiles; the `lint` and `test`
phases are the gates
- bust the Docker layer cache for the gate steps, so `script/cibuild` and
`script/docker` cannot report a green they did not earn (2026-08-09, branch
`cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the
`Dockerfile` copies the tree before its gates, so on an unchanged tree Docker
served them from cache and the build exited 0 having run nothing. Every
`docker build` in `script/` now passes `--no-cache` instead
(https://git.eeqj.de/sneak/sfdupes/issues/95). Run as root, the tests fail
served them from cache and the build exited 0 having run nothing. The fix was
a `CHECK_EPOCH` build argument; since
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
passes `--no-cache` instead. Run as root, the tests fail
`TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)`
the test relies on, so they run as an unprivileged user
the test relies on, so the `test` phase runs them as `nobody`
- check the installed golangci-lint version in `script/bootstrap` instead of
only its presence (2026-08-09, branch `bootstrap-version-check`, closes
https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
+25 -15
View File
@@ -11,6 +11,7 @@ import (
"path/filepath"
"slices"
"strconv"
"time"
"golang.org/x/sys/unix"
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
@@ -40,12 +41,15 @@ const dbDirPerm = 0o755
const lockFilePerm = 0o600
// createTableSQL is the schema applied to a fresh database. Paths are
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime
// holds whole Unix seconds and mtime_nsec the nanoseconds within that
// second.
const createTableSQL = `
CREATE TABLE files (
path BLOB PRIMARY KEY,
size INTEGER NOT NULL,
mtime INTEGER NOT NULL,
mtime_nsec INTEGER NOT NULL,
head TEXT NOT NULL,
tail TEXT NOT NULL,
content TEXT NOT NULL
@@ -61,10 +65,11 @@ CREATE INDEX files_signature ON files (size, head, tail, content)
// upsertSQL inserts one file record, replacing any existing record for
// the same path.
const upsertSQL = `
INSERT INTO files (path, size, mtime, head, tail, content)
VALUES (?, ?, ?, ?, ?, ?)
INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT (path) DO UPDATE SET
size = excluded.size, mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
head = excluded.head, tail = excluded.tail,
content = excluded.content
`
@@ -356,8 +361,8 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
// which is the order of the primary key, so SQLite does not sort.
func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, head, tail, content FROM files "+
"ORDER BY path")
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
"FROM files ORDER BY path")
if err != nil {
return fmt.Errorf("read records: %w", err)
}
@@ -367,16 +372,18 @@ func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
for rows.Next() {
var (
path []byte
sec, nsec int64
r scanRec
)
err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail,
err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail,
&r.content)
if err != nil {
return fmt.Errorf("read record: %w", err)
}
r.path = string(path)
r.mtime = time.Unix(sec, nsec)
fn(r)
}
@@ -466,10 +473,10 @@ func loadDupeRows(ctx context.Context, db *sql.DB,
// values, and skipping the hash columns keeps the scan's in-memory
// index small on multi-million-file databases.
func loadFileMeta(ctx context.Context, db *sql.DB,
fn func(path string, size, mtime int64, hashed bool),
fn func(path string, size int64, mtime time.Time, hashed bool),
) error {
rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, head <> '' FROM files")
"SELECT path, size, mtime, mtime_nsec, head <> '' FROM files")
if err != nil {
return fmt.Errorf("read records: %w", err)
}
@@ -479,16 +486,16 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
for rows.Next() {
var (
path []byte
size, mtime int64
size, sec, nsec int64
hashed int64
)
err = rows.Scan(&path, &size, &mtime, &hashed)
err = rows.Scan(&path, &size, &sec, &nsec, &hashed)
if err != nil {
return fmt.Errorf("read record: %w", err)
}
fn(string(path), size, mtime, hashed != 0)
fn(string(path), size, time.Unix(sec, nsec), hashed != 0)
}
err = rows.Err()
@@ -507,7 +514,7 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
// memory; the rows come ordered by size, head, and tail, so each
// group's rows arrive together.
const contentCandidatesSQL = `
SELECT f.path, f.size, f.mtime, f.head, f.tail, f.content <> ''
SELECT f.path, f.size, f.mtime, f.mtime_nsec, f.head, f.tail, f.content <> ''
FROM files AS f
JOIN (
SELECT size, head, tail
@@ -534,16 +541,19 @@ func loadContentCandidates(ctx context.Context, db *sql.DB,
for rows.Next() {
var (
path []byte
sec, nsec int64
r scanRec
hashed int64
)
err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail, &hashed)
err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail,
&hashed)
if err != nil {
return fmt.Errorf("read record: %w", err)
}
r.path = string(path)
r.mtime = time.Unix(sec, nsec)
fn(r, hashed != 0)
}
@@ -627,8 +637,8 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
defer func() { _ = st.Close() }()
for _, r := range upserts {
_, err = st.ExecContext(ctx,
[]byte(r.path), r.size, r.mtime, r.head, r.tail, r.content)
_, err = st.ExecContext(ctx, []byte(r.path), r.size,
r.mtime.Unix(), r.mtime.Nanosecond(), r.head, r.tail, r.content)
if err != nil {
return fmt.Errorf("upsert %s: %w", r.path, err)
}
+10 -5
View File
@@ -10,6 +10,7 @@ import (
"slices"
"strings"
"testing"
"time"
)
// testDBPath returns a database path inside a fresh temp dir.
@@ -260,10 +261,13 @@ func TestApplyChangesRoundTrip(t *testing.T) {
// written.
recs := []scanRec{
{
size: 2, mtime: 20, head: "h2", tail: "t2", content: "c2",
path: "/a/tab\tnew\nline",
size: 2, mtime: time.Unix(20, 999_999_999), head: "h2", tail: "t2",
content: "c2", path: "/a/tab\tnew\nline",
},
{
size: 1, mtime: time.Unix(10, 0), head: "h1", tail: "t1",
content: "c1", path: "/a/x",
},
{size: 1, mtime: 10, head: "h1", tail: "t1", content: "c1", path: "/a/x"},
}
err := applyChanges(t.Context(), db, recs, nil,
@@ -281,7 +285,8 @@ func TestApplyChangesRoundTrip(t *testing.T) {
// An upsert for an existing path updates in place; a delete
// removes exactly its path.
upd := scanRec{
size: 3, mtime: 30, head: "h3", tail: "t3", content: "c3", path: "/a/x",
size: 3, mtime: time.Unix(30, 0), head: "h3", tail: "t3", content: "c3",
path: "/a/x",
}
err = applyChanges(t.Context(), db, []scanRec{upd},
@@ -308,7 +313,7 @@ func TestApplyChangesBatching(t *testing.T) {
recs := make([]scanRec, 0, n)
for i := range n {
recs = append(recs, scanRec{
size: int64(i), mtime: 1, head: "h", tail: "t",
size: int64(i), mtime: time.Unix(1, 0), head: "h", tail: "t",
path: fmt.Sprintf("/batch/%07d", i),
})
}
+2 -1
View File
@@ -7,6 +7,7 @@ import (
"io"
"os"
"strings"
"time"
)
// ioBufSize is the buffer size for the buffered stdout writers.
@@ -21,7 +22,7 @@ const minGroupSize = 2
// only and used by scan for change detection.
type scanRec struct {
size int64
mtime int64
mtime time.Time
head string
tail string
content string
+9 -2
View File
@@ -11,6 +11,7 @@ import (
"slices"
"strings"
"testing"
"time"
)
// awkwardDir is a directory name holding every byte the reports escape.
@@ -313,8 +314,14 @@ func TestDupeGroupsMtimeExcluded(t *testing.T) {
// mtime is informational only; records differing only in mtime
// still group together.
recs := []scanRec{
{size: 9, mtime: 100, head: "h", tail: "t", content: "c", path: "/m/1"},
{size: 9, mtime: 200, head: "h", tail: "t", content: "c", path: "/m/2"},
{
size: 9, mtime: time.Unix(100, 0), head: "h", tail: "t",
content: "c", path: "/m/1",
},
{
size: 9, mtime: time.Unix(200, 0), head: "h", tail: "t",
content: "c", path: "/m/2",
},
}
groups := dupeGroupsOf(t, recs)
+18 -7
View File
@@ -17,6 +17,7 @@ import (
"strings"
"sync"
"syscall"
"time"
)
// The duplicate ladder (see hashSignature and README "Duplicate
@@ -68,7 +69,7 @@ var errInterrupted = errors.New("scan interrupted")
type fileRec struct {
path string
size int64
mtime int64
mtime time.Time
dev uint64
ino uint64
}
@@ -79,7 +80,7 @@ type fileRec struct {
// they would dominate the scan's memory.
type fileMeta struct {
size int64
mtime int64
mtime time.Time
hashed bool
}
@@ -369,7 +370,7 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
s.existing = make(map[string]fileMeta)
return loadFileMeta(ctx, s.db,
func(path string, size, mtime int64, hashed bool) {
func(path string, size int64, mtime time.Time, hashed bool) {
prog.increment()
if underAnyRoot(path, roots) {
@@ -415,7 +416,7 @@ func (s *scanState) walkPhase(
old, ok := s.existing[ev.rec.path]
switch {
case !ok || old.size != ev.rec.size || old.mtime < ev.rec.mtime:
case !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime):
changed = append(changed, ev.rec)
case old.hashed:
delete(s.existing, ev.rec.path)
@@ -807,7 +808,7 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
}
if !fi.Mode().IsRegular() || fi.Size() != r.size ||
fi.ModTime().Unix() > r.mtime {
mtimeAfter(fi.ModTime(), r.mtime) {
return fileRec{}, false, nil
}
@@ -818,6 +819,16 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
}, true, nil
}
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under
// one of them.
func underAnyRoot(path string, roots []string) bool {
@@ -966,7 +977,7 @@ func seedRoot(ctx context.Context, root string,
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: root,
size: fi.Size(),
mtime: fi.ModTime().Unix(),
mtime: fi.ModTime(),
dev: dev,
ino: ino,
}})
@@ -1124,7 +1135,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: p,
size: info.Size(),
mtime: info.ModTime().Unix(),
mtime: info.ModTime(),
dev: dev,
ino: ino,
}})
+227 -7
View File
@@ -20,6 +20,8 @@ import (
"syscall"
"testing"
"time"
"golang.org/x/sys/unix"
)
// writeFile creates a file with the given content and returns its path.
@@ -581,6 +583,65 @@ func TestScanContentHashedStalePartners(t *testing.T) {
}
}
// TestScanContentSameSecondRewrite is TestScanContentStalePartners for
// a stored file rewritten in place at the same size with an mtime later
// in the same second than recorded: the file counts as changed, so
// neither it nor its match inside the operand is read.
func TestScanContentSameSecondRewrite(t *testing.T) {
t.Parallel()
db := openTestDB(t)
dirA := t.TempDir()
changed := sparseFileWithoutMatch(t, dirA, "changed", headTailMin)
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(changed, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dirA)
before := dbRecords(t, db)
// Rewrite one byte in place, keeping the size.
pokeAt(t, changed, headTailMin/2, []byte{1})
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(changed, later, later)
if err != nil {
t.Fatal(err)
}
dirB := t.TempDir()
sparseFile(t, dirB, "changed-copy", headTailMin)
st := syncTree(t, db, dirB)
if st != (scanStats{walked: 1, added: 1}) {
t.Errorf("stats = %+v, want 1 added and nothing skipped", st)
}
recs := dbRecords(t, db)
for _, r := range recs {
if r.content != "" {
t.Errorf("%s: content = %q, want none: its only match is stale",
r.path, r.content)
}
}
for _, old := range before {
if r := recordByPath(t, recs, old.path); r != old {
t.Errorf("record = %+v, want it left as %+v", r, old)
}
}
if groups := dupeGroups(t, db); len(groups) != 0 {
t.Errorf("groups = %+v, want none", groups)
}
}
// TestScanContentReadFailure checks that a failed content read is
// counted as skipped and leaves the record without a content hash, and
// that a later scan tries the read again.
@@ -783,8 +844,8 @@ func TestWalk(t *testing.T) {
t.Errorf("%s: size = %d, want 1..3", r.path, r.size)
}
if r.mtime <= 0 {
t.Errorf("%s: mtime = %d, want positive", r.path, r.mtime)
if r.mtime.Unix() <= 0 {
t.Errorf("%s: mtime = %v, want after 1970", r.path, r.mtime)
}
}
}
@@ -1285,11 +1346,172 @@ func TestSyncScanMtimeBump(t *testing.T) {
t.Fatalf("mtime-bump stats = %+v, want 1 updated", st)
}
if r := recordByPath(t, dbRecords(t, db), a); r.mtime != future.Unix() {
t.Fatalf("mtime = %d, want %d", r.mtime, future.Unix())
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(future) {
t.Fatalf("mtime = %v, want %v", r.mtime, future)
}
}
// assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
data []byte,
) {
t.Helper()
r := recordByPath(t, dbRecords(t, db), path)
if want := hexSum(data); r.head != want || r.tail != want ||
r.content != want {
t.Fatalf("head, tail, content = %q, %q, %q, want %q for each",
r.head, r.tail, r.content, want)
}
}
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
// size with an mtime later in the same second as the recorded one: the
// next scan must notice the change and re-hash the file.
func TestSyncScanSameSecondRewrite(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
writeFile(t, dir, "b.bin", pattern(2, 500))
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(a, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(a, later, later)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
// TestSyncScanOperandSameSecondRewrite is TestSyncScanSameSecondRewrite
// for files given to scan as operands, which scan stats without reading
// their directory.
func TestSyncScanOperandSameSecondRewrite(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
b := writeFile(t, dir, "b.bin", pattern(2, 500))
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(a, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, a, b)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(a, later, later)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, a, b)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
// TestSyncScanRewriteAfter2262 runs assertLateRewriteRehashed with an
// mtime after 2262, a time too late to count in nanoseconds in an int64.
func TestSyncScanRewriteAfter2262(t *testing.T) {
t.Parallel()
assertLateRewriteRehashed(t, time.Date(2300, 1, 2, 3, 4, 5, 0, time.UTC))
}
// TestSyncScanRewritePastTimeLimit runs assertLateRewriteRehashed with an
// mtime one second past the latest a time.Time holds without wrapping it
// to a time far in the past.
func TestSyncScanRewritePastTimeLimit(t *testing.T) {
t.Parallel()
assertLateRewriteRehashed(t, time.Unix(9223371974719179008, 0))
}
// assertLateRewriteRehashed scans a directory, rewrites a file in it in
// place at the same size, sets its mtime to late, and fails unless the
// next scan re-hashes the file. It skips where late does not fit the
// platform's timespec or the filesystem does not store it.
func assertLateRewriteRehashed(t *testing.T, late time.Time) {
t.Helper()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
writeFile(t, dir, "b.bin", pattern(2, 500))
syncTree(t, db, dir)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
// os.Chtimes cannot set such a time: it converts through UnixNano.
ts, err := unix.TimeToTimespec(late)
if err != nil {
t.Skipf("an mtime %d seconds after 1970 does not fit this platform's "+
"timespec: %v", late.Unix(), err)
}
err = unix.UtimesNano(a, []unix.Timespec{ts, ts})
if err != nil {
t.Fatal(err)
}
fi, err := os.Lstat(a)
if err != nil {
t.Fatal(err)
}
if fi.ModTime().Unix() != late.Unix() {
t.Skipf("the filesystem stored the mtime as %d seconds after 1970, "+
"not %d", fi.ModTime().Unix(), late.Unix())
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
func TestSyncScanAddRemove(t *testing.T) {
t.Parallel()
@@ -1335,9 +1557,7 @@ func TestSyncScanSizeChange(t *testing.T) {
writeFile(t, dir, "f", pattern(1, 200))
mt := time.Unix(old.mtime, 0)
err := os.Chtimes(p, mt, mt)
err := os.Chtimes(p, old.mtime, old.mtime)
if err != nil {
t.Fatal(err)
}
+84 -16
View File
@@ -1,13 +1,22 @@
#!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent; assumes nothing is present (not git, make, or
# go). Base tooling comes from nix, apt, brew, or apk (detected in that
# order). golangci-lint and prettier are never installed: they run via
# docker only (script/lint, script/fmt, script/fmt-check).
# this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present. Node is
# used directly if installed; otherwise it is installed at a pinned
# version via nvm (installing nvm itself first, from a hash-verified
# release archive, never curl | sh).
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-06
NODE_VERSION="22.17.0"
NVM_VERSION="0.40.3"
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
YARN_VERSION="1.22.22"
PKGMGR=""
SUDO=""
APT_UPDATED=""
@@ -40,6 +49,7 @@ pkg_install() {
case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;;
apt)
# Package lists may be empty (fresh images); refresh once per run.
if [ -z "$APT_UPDATED" ]; then
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get update
APT_UPDATED=1
@@ -55,24 +65,82 @@ missing() {
! command -v "$1" >/dev/null 2>&1
}
# verify_sha256 <file> <expected-hash>
verify_sha256() {
if command -v sha256sum >/dev/null 2>&1; then
actual="$(sha256sum "$1" | cut -d' ' -f1)"
else
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
fi
if [ "$actual" != "$2" ]; then
echo "bootstrap: sha256 mismatch for $1" >&2
echo " expected: $2" >&2
echo " actual: $actual" >&2
exit 1
fi
}
# nvm is a bash script; run a command in a bash with nvm loaded
nvm_sh() {
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
}
ensure_nvm() {
[ -s "$HOME/.nvm/nvm.sh" ] && return 0
# nvm prerequisites; nvm itself requires bash
if missing bash; then pkg_install bash bash bash bash; fi
if missing curl; then pkg_install curl curl curl curl; fi
if missing git; then pkg_install git git git git; fi
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/nvm.tar.gz" \
"https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
mkdir -p "$HOME/.nvm"
tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
rm -rf "$tmp"
}
ensure_node() {
if ! missing node; then return 0; fi
ensure_nvm
nvm_sh "nvm install $NODE_VERSION"
}
ensure_yarn() {
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable
corepack prepare "yarn@$YARN_VERSION" --activate
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
else
npm install -g "yarn@$YARN_VERSION"
fi
}
install_js_deps() {
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
yarn install --frozen-lockfile"
else
yarn install --frozen-lockfile
fi
}
main() {
cd "$ROOT"
# Deliberately unpinned, so presence is the whole check: go.mod
# governs the Go version, and reproducible builds run in the
# digest-pinned Docker images.
if missing git; then pkg_install git git git git; fi
if missing make; then pkg_install gnumake make make make; fi
if missing git; then pkg_install git git git git; fi
# Go builds the binary and runs gofmt. Presence is the whole check:
# go.mod names the Go version, and the tests and the linter run in
# digest-pinned images.
if missing go; then pkg_install go golang go go; fi
# Warn, do not fail: only the targets named below, and the
# pre-commit hook, need docker.
if missing docker; then
echo "bootstrap: WARNING: docker not found; make test, make lint," >&2
echo "bootstrap: make fmt, make fmt-check, make check and" >&2
echo "bootstrap: make docker require it." >&2
fi
ensure_node
ensure_yarn
install_js_deps
go mod download
echo "bootstrap complete"
+3 -1
View File
@@ -1,6 +1,8 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files.
# extension to scripts-to-rule-them-all. test and lint are Docker
# phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+7 -5
View File
@@ -14,15 +14,17 @@ main() {
cd "$ROOT"
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
# The version and the tag each get their own line: a failing
# command substitution inside an argument does not trip `set -e`,
# so the inline form degrades silently to an empty constant. The
# VERSION build argument takes precedence over the version a build
# stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
-t "$tag" .
}
main "$@"
+7 -5
View File
@@ -10,15 +10,17 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
# The version and the tag each get their own line: a failing
# command substitution inside an argument does not trip `set -e`,
# so the inline form degrades silently to an empty constant. The
# VERSION build argument takes precedence over the version a build
# stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
-t "$tag" .
}
main "$@"
+22 -13
View File
@@ -1,23 +1,32 @@
#!/bin/sh
# script/fmt: format all files (writes): the Go sources with gofmt, the
# Markdown with prettier. prettier is never installed on the host: it
# runs from the Dockerfile's prettier stage with the repository mounted,
# as the calling user so the files it rewrites keep their owner. The
# build passes --no-cache, as every docker build in script/ does. The tag
# makes each build replace the previous image instead of leaving another
# one behind.
# script/fmt: format all files (writes).
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() {
cd "$ROOT"
gofmt -s -w .
image="$("$SCRIPT_DIR/projectname")-prettier"
docker build -q --no-cache --target prettier -t "$image" . >/dev/null
docker run --rm --user "$(id -u):$(id -g)" -v "$ROOT:/src" "$image" \
prettier --write '**/*.md' --tab-width 4 --prose-wrap always
run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always
}
main "$@"
+26 -15
View File
@@ -1,19 +1,35 @@
#!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing. gofmt and prettier both run
# every time and each reports its own failure, so the output says which
# one failed.
# script/fmt-check: check formatting (read-only).
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt-check: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() {
cd "$ROOT"
status=0
# Under set -e a bare assignment would end the script when gofmt
# fails (a Go file it cannot parse), and prettier would never run.
# gofmt and prettier both run every time, so the output names each
# one that fails. Under set -e a bare assignment would end the
# script when gofmt fails (a Go file it cannot parse).
if ! files="$(gofmt -s -l .)"; then
echo "gofmt: failed; see its errors above" >&2
status=1
@@ -24,14 +40,9 @@ main() {
status=1
fi
# Same image as script/fmt; see there.
image="$("$SCRIPT_DIR/projectname")-prettier"
docker build -q --no-cache --target prettier -t "$image" . >/dev/null
if ! docker run --rm -v "$ROOT:/src:ro" "$image" \
prettier --check '**/*.md' --tab-width 4 --prose-wrap always; then
echo "prettier: Markdown not formatted; run make fmt" >&2
# run_yarn ends in exec; the subshell returns here afterwards.
(run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always) ||
status=1
fi
exit "$status"
}
+3 -7
View File
@@ -1,19 +1,15 @@
#!/bin/sh
# script/install-precommit: install the git pre-commit hook that runs
# script/precommit. Our own extension to scripts-to-rule-them-all.
# Hooks are shared between the main checkout and all worktrees, so
# resolve the common git dir instead of assuming .git is a directory.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
hooks_dir="$(git rev-parse --git-common-dir)/hooks"
mkdir -p "$hooks_dir"
hook="$hooks_dir/pre-commit"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > "$hook"
chmod +x "$hook"
hook=".git/hooks/pre-commit"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
echo "pre-commit hook installed: runs script/precommit"
}
+3 -3
View File
@@ -6,8 +6,8 @@
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
# that did not run. --output type=cacheonly writes no image, since
# nothing uses one.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -17,7 +17,7 @@ main() {
cd "$ROOT"
docker build --no-cache \
--target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
--output type=cacheonly .
}
main "$@"
+2 -2
View File
@@ -1,13 +1,13 @@
#!/bin/sh
# script/precommit: run by the git pre-commit hook; fails the commit if
# checks fail. Our own extension to scripts-to-rule-them-all. Go extra:
# go mod tidy must be a no-op before the checks run.
# checks fail. Our own extension to scripts-to-rule-them-all.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
# Go extra: go mod tidy must be a no-op before the checks run.
cd "$ROOT"
go mod tidy
if ! git diff --exit-code -- go.mod go.sum; then
+1 -1
View File
@@ -1,6 +1,6 @@
#!/bin/sh
# script/setup: set up the repo for development after a fresh clone:
# installs dependencies (script/bootstrap) and the git pre-commit hook.
# installs dependencies and the git pre-commit hook.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+3 -3
View File
@@ -2,8 +2,8 @@
# script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
# named, and --no-cache because a cached test layer is a test that did
# not run. --output type=cacheonly writes no image, since nothing uses one.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -13,7 +13,7 @@ main() {
cd "$ROOT"
docker build --no-cache \
--target test \
-t "$("$SCRIPT_DIR/projectname")-test" .
--output type=cacheonly .
}
main "$@"