Fold the August fleet findings into the policies, or drop them #62

New Issue

2026-09-25T12:25:17+02:00

clawbot commented

2026-09-25 12:25:17 +02:00

Moved from the project-management repo's TODO.md (sneak/project-management#9), where it sat since 2026-08-09 as "cross-cutting findings worth remembering". Three of them became #25, #26 and #27, all closed. The rest were never folded into the policies. Git worktrees have since been banned outright, so the worktree items only explain history.

Definition of done

Each finding below is either written into the policies or dropped here with the reason.

The findings, as recorded on 2026-08-09

Cross-cutting findings worth remembering:

The pre-migration .golangci.yml (sha256 33ba2bf7...) declared
version: "2" with v1 layout, so golangci-lint silently fell back to defaults
and every threshold was ignored — any repo still on it has been reporting
meaningless 0 issues.. Detect with golangci-lint config verify, but NEVER
wire that into make lint or CI: it fetches its schema over an unpinned live
HTTPS call.
Pinning golangci-lint by Docker image digest (@sha256:...) satisfies the
hash-pinning policy. Demanding the git commit c0d3ddc9... from a repo with a
separate pinned lint stage is a phantom violation — the top-sdlc-manager's
launch prompts got this wrong and several managers correctly refused it.
The gomodguard deprecation under v2.12 belongs upstream: filed as prompts
#25, since the canonical config is untouchable in consuming repos.
ORDERING HAZARD, fleet-wide, and the rule is BROADER than "repos with no
.dockerignore": ANY directory in the Docker build context that churns
accidentally protects a repo, because it invalidates COPY . . and forces the
check layers to re-run. Tightening .dockerignore removes that protection.
Two known churning directories:
- .git — churns on nearly every git operation. Protects repos that have no
  .dockerignore at all; the canonical file excludes it.
- .claude/ — holds worktrees/, created and destroyed constantly wherever
  agents run. This one hits repos that ALREADY exclude .git and therefore
  look unaffected. prompts #27 adds the exclusion.
CONFIRMED by controlled experiment on rfscan 2026-08-09 (isolated copy,
one variable per pair, only files inside .git touched): with no
.dockerignore a .git-only change forces a real ~10s run; with .git
excluded the identical change is a 0.37s cache hit, and the toggle reverses
cleanly three times. ABSENCE OF A .dockerignore IS NOT A SCREENING TEST —
a repo that has always excluded .git never had the protection and is
exposed RIGHT NOW. EVERY repo needs the cache-bust regardless of its
.dockerignore state. A cache hit is SUB-SECOND: a "warm" run of a minute
or more is a real run, and the question becomes what else in the context is
mutating (.claude/worktrees being the usual answer). Measure in an
isolated copy with zero git activity between paired runs. Where a repo has
NO .dockerignore, the canonical file and the cache-bust must land TOGETHER
or cache-bust-first — never .dockerignore-first, and they are sometimes
atomic in one PR. Order per repo: land the ARG CHECK_EPOCH fix (prompts
#26) → verify two consecutive runs execute the checks AND that the bootstrap
layer still caches → land the .dockerignore tightening (prompts #27) →
RE-VERIFY, because the context changed underneath the earlier verification.
That last step is the one most likely to be skipped, and skipping it is the
same mistake as the lora.vegas outage: validating against one environment
and assuming the result transfers to a changed one.
NESTED AGENT WORKTREES POLLUTE TOOLS THAT WALK THE FILESYSTEM — but which
tools, exactly, depends on their DISCOVERY semantics, not on being a "test
runner". Corrected by bsfirehose after an over-broad first statement:
- SAFE: go test ./.... The go tool ignores directories beginning with .
  or _ (go help packages), so it never descends into
  .claude/worktrees/. No N+1 inflation; a 30s timeout measures what it
  claims to.
- UNSAFE: gofmt -l ., and anything else doing a plain filesystem walk. It
  descends into nested worktrees and can fail make check on files that are
  not part of the repo at all. Tracked as bsfirehose #24.
- UNSAFE: JS test runners. quak measured 1050 tests against a real 210 (18
  files x 5) — vitest's default exclude does not cover .claude/ and it
  does not read .gitignore for discovery. There the wall-clock inflation
  is real and does distort a timed gate. Reviewers inside their own
  worktrees are unaffected either way. This only shows in the SHARED clone —
  exactly where a manager does post-merge verification and is most likely to
  trust the number.
NEVER docker builder prune ON THIS HOST. It is shared mutable state across
~18 concurrent sessions. One subagent destroyed ~41 GB of shared BuildKit
cache on 2026-08-09 while trying to prove a build was uncached. To prove that,
scope the invalidation: docker build --no-cache on the one image, or
--no-cache-filter=<stage> on the check stage. Note the prohibition must
travel WITH the "a green can lie to you" caution, not after it — the moment an
agent is told the cache is untrustworthy, destroying it is the obvious next
thought, which is exactly how this happened.
- Fallout to expect after any prune: cold-cache timeout flakes that look
  like new bugs (netwatch saw timeout 30 go test ./... die on an empty Go
  build cache, then pass in 11s on retry), and cache measurements taken
  across the boundary being invalid pairs.
DISPATCH RULE: idle in ListAgents does NOT mean stalled. A repo-manager is
idle between turns — having just reported, or waiting on a respawned
reviewer. Nudging on that signal alone produces messages asserting things that
are not true (it happened twice on 2026-08-09, to eight sessions and then to
two more, and every one pushed back with evidence). Before nudging, re-read
the repo's tracker; and phrase the nudge as a question about state rather than
a claim about it.
DISPATCH RULE for the top-sdlc-manager: resume-or-dispatch is an EITHER/OR.
Sending a new requirement to an already-running agent AND spawning a rework
agent for the same task puts two agents on one branch. It happened on
cattbox 2026-08-07; no conflicting history resulted only because the
no-force-push rule held. Pick one before acting.
When fixing the cache hole, verify the BOOTSTRAP layer still caches, not just
that the check re-runs. A blanket --no-cache "fixes" the hole while turning
a ~10s check into a full toolchain reinstall on every run.
A check that passes for an accidental reason fails silently the moment the
accident is tidied up. Three instances so far: .git churn forcing real runs;
netwatch's schema-invalid .golangci.yml making 0 issues. meaningless;
netwatch's root make check genuinely passing while never touching the
backend at all.
Evidence standard, in descending strength: a NEGATIVE CONTROL (a cached layer
cannot produce a specifically predicted failure, and cannot fail then pass
after a fix) beats layer inspection, which beats wall-clock. A green with
neither duration nor a negative control is UNVERIFIED, not passed. Related
discipline: "corroborated by a subagent" is not "verified" — say which.
"Pin by hash" is not "bump to the latest major". lora.vegas PR #17 passed
two independent reviews and still broke production, because an
upload-artifact v3->v4 bump assumed a protocol this Gitea does not serve. A
CI path that cannot be executed pre-merge cannot be validated by review at all
— use a temporary branch trigger plus if: github.ref_name == 'main' on
publishing jobs, and read verdicts from the commit-status API.

Model: opus-5-5

Moved from the project-management repo's `TODO.md` (https://git.eeqj.de/sneak/project-management/issues/9), where it sat since 2026-08-09 as "cross-cutting findings worth remembering". Three of them became https://git.eeqj.de/sneak/prompts/issues/25, https://git.eeqj.de/sneak/prompts/issues/26 and https://git.eeqj.de/sneak/prompts/issues/27, all closed. The rest were never folded into the policies. Git worktrees have since been banned outright, so the worktree items only explain history. ## Definition of done - Each finding below is either written into the policies or dropped here with the reason. ## The findings, as recorded on 2026-08-09 Cross-cutting findings worth remembering: - The pre-migration `.golangci.yml` (sha256 `33ba2bf7...`) declared `version: "2"` with v1 layout, so golangci-lint silently fell back to defaults and every threshold was ignored — any repo still on it has been reporting meaningless `0 issues.`. Detect with `golangci-lint config verify`, but NEVER wire that into `make lint` or CI: it fetches its schema over an unpinned live HTTPS call. - Pinning golangci-lint by Docker image digest (`@sha256:...`) satisfies the hash-pinning policy. Demanding the git commit `c0d3ddc9...` from a repo with a separate pinned lint stage is a phantom violation — the top-sdlc-manager's launch prompts got this wrong and several managers correctly refused it. - The `gomodguard` deprecation under v2.12 belongs upstream: filed as `prompts` #25, since the canonical config is untouchable in consuming repos. - ORDERING HAZARD, fleet-wide, and the rule is BROADER than "repos with no `.dockerignore`": ANY directory in the Docker build context that churns accidentally protects a repo, because it invalidates `COPY . .` and forces the check layers to re-run. Tightening `.dockerignore` removes that protection. Two known churning directories: - `.git` — churns on nearly every git operation. Protects repos that have no `.dockerignore` at all; the canonical file excludes it. - `.claude/` — holds `worktrees/`, created and destroyed constantly wherever agents run. This one hits repos that ALREADY exclude `.git` and therefore look unaffected. `prompts` #27 adds the exclusion. CONFIRMED by controlled experiment on `rfscan` 2026-08-09 (isolated copy, one variable per pair, only files inside `.git` touched): with no `.dockerignore` a `.git`-only change forces a real ~10s run; with `.git` excluded the identical change is a 0.37s cache hit, and the toggle reverses cleanly three times. ABSENCE OF A `.dockerignore` IS NOT A SCREENING TEST — a repo that has always excluded `.git` never had the protection and is exposed RIGHT NOW. EVERY repo needs the cache-bust regardless of its `.dockerignore` state. A cache hit is SUB-SECOND: a "warm" run of a minute or more is a real run, and the question becomes what else in the context is mutating (`.claude/worktrees` being the usual answer). Measure in an isolated copy with zero git activity between paired runs. Where a repo has NO `.dockerignore`, the canonical file and the cache-bust must land TOGETHER or cache-bust-first — never `.dockerignore`-first, and they are sometimes atomic in one PR. Order per repo: land the `ARG CHECK_EPOCH` fix (`prompts` #26) → verify two consecutive runs execute the checks AND that the bootstrap layer still caches → land the `.dockerignore` tightening (`prompts` #27) → RE-VERIFY, because the context changed underneath the earlier verification. That last step is the one most likely to be skipped, and skipping it is the same mistake as the lora.vegas outage: validating against one environment and assuming the result transfers to a changed one. - NESTED AGENT WORKTREES POLLUTE TOOLS THAT WALK THE FILESYSTEM — but which tools, exactly, depends on their DISCOVERY semantics, not on being a "test runner". Corrected by `bsfirehose` after an over-broad first statement: - SAFE: `go test ./...`. The go tool ignores directories beginning with `.` or `_` (`go help packages`), so it never descends into `.claude/worktrees/`. No N+1 inflation; a 30s timeout measures what it claims to. - UNSAFE: `gofmt -l .`, and anything else doing a plain filesystem walk. It descends into nested worktrees and can fail `make check` on files that are not part of the repo at all. Tracked as `bsfirehose` #24. - UNSAFE: JS test runners. `quak` measured 1050 tests against a real 210 (18 files x 5) — vitest's default exclude does not cover `.claude/` and it does not read `.gitignore` for discovery. There the wall-clock inflation is real and does distort a timed gate. Reviewers inside their own worktrees are unaffected either way. This only shows in the SHARED clone — exactly where a manager does post-merge verification and is most likely to trust the number. - NEVER `docker builder prune` ON THIS HOST. It is shared mutable state across ~18 concurrent sessions. One subagent destroyed ~41 GB of shared BuildKit cache on 2026-08-09 while trying to prove a build was uncached. To prove that, scope the invalidation: `docker build --no-cache` on the one image, or `--no-cache-filter=<stage>` on the check stage. Note the prohibition must travel WITH the "a green can lie to you" caution, not after it — the moment an agent is told the cache is untrustworthy, destroying it is the obvious next thought, which is exactly how this happened. - Fallout to expect after any prune: cold-cache timeout flakes that look like new bugs (netwatch saw `timeout 30 go test ./...` die on an empty Go build cache, then pass in 11s on retry), and cache measurements taken across the boundary being invalid pairs. - DISPATCH RULE: `idle` in `ListAgents` does NOT mean stalled. A repo-manager is `idle` between turns — having just reported, or waiting on a respawned reviewer. Nudging on that signal alone produces messages asserting things that are not true (it happened twice on 2026-08-09, to eight sessions and then to two more, and every one pushed back with evidence). Before nudging, re-read the repo's tracker; and phrase the nudge as a question about state rather than a claim about it. - DISPATCH RULE for the top-sdlc-manager: resume-or-dispatch is an EITHER/OR. Sending a new requirement to an already-running agent AND spawning a rework agent for the same task puts two agents on one branch. It happened on `cattbox` 2026-08-07; no conflicting history resulted only because the no-force-push rule held. Pick one before acting. - When fixing the cache hole, verify the BOOTSTRAP layer still caches, not just that the check re-runs. A blanket `--no-cache` "fixes" the hole while turning a ~10s check into a full toolchain reinstall on every run. - A check that passes for an accidental reason fails silently the moment the accident is tidied up. Three instances so far: `.git` churn forcing real runs; netwatch's schema-invalid `.golangci.yml` making `0 issues.` meaningless; netwatch's root `make check` genuinely passing while never touching the backend at all. - Evidence standard, in descending strength: a NEGATIVE CONTROL (a cached layer cannot produce a specifically predicted failure, and cannot fail then pass after a fix) beats layer inspection, which beats wall-clock. A green with neither duration nor a negative control is UNVERIFIED, not passed. Related discipline: "corroborated by a subagent" is not "verified" — say which. - "Pin by hash" is not "bump to the latest major". `lora.vegas` PR #17 passed two independent reviews and still broke production, because an `upload-artifact` v3->v4 bump assumed a protocol this Gitea does not serve. A CI path that cannot be executed pre-merge cannot be validated by review at all — use a temporary branch trigger plus `if: github.ref_name == 'main'` on publishing jobs, and read verdicts from the commit-status API. Model: opus-5-5

clawbot commented

2026-10-03 14:19:54 +02:00

Plan. One PR plus one disposition comment on this issue.

For each finding, decide fold or drop against the canonical documents on next:

Drop a finding that the policies already state (for example, pruning the shared build cache is already forbidden, and the golangci-lint image digest is already the only pin), that a later ruling replaced (every docker build in script/ now passes --no-cache, so the cache-bust ordering hazard and the bootstrap-layer caching check no longer apply), that only explains git worktrees (now banned), or that is about how agents dispatch and nudge each other rather than about a repository (those belong in the agent definitions, not here).
Fold only a rule a repository must follow that the documents do not yet state, in at most two plain sentences, in the section where a reader would look for it. Likely candidates: never wiring golangci-lint config verify into make lint or CI because it fetches its schema over the network; checking a gate by planting a defect and watching it fail, not by a fast green; and a CI step that only runs on main cannot be checked by review, so try it on a branch trigger first.

Done when every finding has one line in the disposition comment (fold: where; drop: why), the PR adds only the folded sentences, and make check passes.

Model: opus-5-5

Plan. One PR plus one disposition comment on this issue. For each finding, decide fold or drop against the canonical documents on `next`: - Drop a finding that the policies already state (for example, pruning the shared build cache is already forbidden, and the golangci-lint image digest is already the only pin), that a later ruling replaced (every `docker build` in `script/` now passes `--no-cache`, so the cache-bust ordering hazard and the bootstrap-layer caching check no longer apply), that only explains git worktrees (now banned), or that is about how agents dispatch and nudge each other rather than about a repository (those belong in the agent definitions, not here). - Fold only a rule a repository must follow that the documents do not yet state, in at most two plain sentences, in the section where a reader would look for it. Likely candidates: never wiring `golangci-lint config verify` into `make lint` or CI because it fetches its schema over the network; checking a gate by planting a defect and watching it fail, not by a fast green; and a CI step that only runs on `main` cannot be checked by review, so try it on a branch trigger first. Done when every finding has one line in the disposition comment (fold: where; drop: why), the PR adds only the folded sentences, and `make check` passes. Model: opus-5-5

clawbot referenced this issue from a commit

2026-10-04 11:36:03 +02:00

Fold the August fleet findings into the policies, or drop them (closes #62)

clawbot referenced a pull request that will close this issue

2026-10-04 11:36:09 +02:00

Fold the August fleet findings into the policies, or drop them (closes #62) #99

clawbot commented

2026-10-04 11:36:20 +02:00

Disposition of each finding, in the order recorded. PR: #99

.golangci.yml declaring version: "2" with the v1 layout: drop. sneak ruled on #40 (2026-08-10) that there is no config check step and the config is assumed valid; a vendored .golangci.yml stays byte-identical to the canonical copy.
golangci-lint pinned by image digest, not git commit: drop. Already stated: the image digest is the only pin, since no repo installs golangci-lint on the host.
gomodguard deprecation belongs upstream: drop. Done in #25, and the policies already send linter config changes to the canonical copy.
Ordering hazard between the cache-bust and .dockerignore: drop. Replaced: every docker build in script/ passes --no-cache, so what changes in the build context no longer decides whether the checks run. .claude is already excluded.
Nested agent worktrees and tools that walk the filesystem: drop. Only explains git worktrees, which are now banned.
Never docker builder prune: drop. Already stated in the --no-cache paragraph, beside the warning that a fast green is a cache hit. The fallout after a prune concerns the shared host, not a repository.
idle does not mean stalled: drop. About how agents nudge each other; belongs in the agent definitions.
Resume or dispatch, never both: drop. About how agents are dispatched; belongs in the agent definitions.
Check that the bootstrap layer still caches: drop. Replaced: --no-cache everywhere accepts the full rebuild, as the Gitea Actions workflow paragraph says.
A check that passes for an accidental reason: fold. Covered by the sentence added for finding 11, which catches a check that never ran and one that covers less than it claims.
Evidence standard: fold. The end of the --no-cache paragraph now says a new or changed check is proven by planting a defect it must catch and watching the run fail. The point about a subagent's corroboration is about how agents report, not about a repository: dropped.
Pin by hash is not a version bump; a CI path that cannot run before merge: fold. The Gitea Actions workflow paragraph now says a change to a separate workflow limited to main by a branches list under on: push is tried by adding the feature branch to that list and removing it again before merging, with any job that publishes kept behind if: github.ref_name == 'main'. Judgement call: the version bump gets no rule of its own, since it broke production only because nothing could run that workflow before merge. Reading results from the commit-status API is agent practice: dropped.

Model: opus-5-5

Disposition of each finding, in the order recorded. PR: https://git.eeqj.de/sneak/prompts/pulls/99 1. `.golangci.yml` declaring `version: "2"` with the v1 layout: drop. sneak ruled on https://git.eeqj.de/sneak/prompts/issues/40 (2026-08-10) that there is no config check step and the config is assumed valid; a vendored `.golangci.yml` stays byte-identical to the canonical copy. 2. golangci-lint pinned by image digest, not git commit: drop. Already stated: the image digest is the only pin, since no repo installs golangci-lint on the host. 3. `gomodguard` deprecation belongs upstream: drop. Done in https://git.eeqj.de/sneak/prompts/issues/25, and the policies already send linter config changes to the canonical copy. 4. Ordering hazard between the cache-bust and `.dockerignore`: drop. Replaced: every `docker build` in `script/` passes `--no-cache`, so what changes in the build context no longer decides whether the checks run. `.claude` is already excluded. 5. Nested agent worktrees and tools that walk the filesystem: drop. Only explains git worktrees, which are now banned. 6. Never `docker builder prune`: drop. Already stated in the `--no-cache` paragraph, beside the warning that a fast green is a cache hit. The fallout after a prune concerns the shared host, not a repository. 7. `idle` does not mean stalled: drop. About how agents nudge each other; belongs in the agent definitions. 8. Resume or dispatch, never both: drop. About how agents are dispatched; belongs in the agent definitions. 9. Check that the bootstrap layer still caches: drop. Replaced: `--no-cache` everywhere accepts the full rebuild, as the Gitea Actions workflow paragraph says. 10. A check that passes for an accidental reason: fold. Covered by the sentence added for finding 11, which catches a check that never ran and one that covers less than it claims. 11. Evidence standard: fold. The end of the `--no-cache` paragraph now says a new or changed check is proven by planting a defect it must catch and watching the run fail. The point about a subagent's corroboration is about how agents report, not about a repository: dropped. 12. Pin by hash is not a version bump; a CI path that cannot run before merge: fold. The Gitea Actions workflow paragraph now says a change to a separate workflow limited to `main` by a `branches` list under `on: push` is tried by adding the feature branch to that list and removing it again before merging, with any job that publishes kept behind `if: github.ref_name == 'main'`. Judgement call: the version bump gets no rule of its own, since it broke production only because nothing could run that workflow before merge. Reading results from the commit-status API is agent practice: dropped. Model: opus-5-5

clawbot referenced this issue

2026-10-04 11:54:27 +02:00

Fold the August fleet findings into the policies, or drop them (closes #62) #99

clawbot referenced this issue from a commit

2026-10-04 12:05:33 +02:00

Fold the August fleet findings into the policies, or drop them (closes #62)

clawbot referenced this issue

2026-10-04 12:06:18 +02:00

Fold the August fleet findings into the policies, or drop them (closes #62) #99

clawbot referenced this issue

2026-10-04 12:18:48 +02:00

Fold the August fleet findings into the policies, or drop them (closes #62) #99