issue-to-pr workers break the no-scripted-search-and-replace and no-raw-go-tools rules at a measured 3-in-4 rate #51

Closed
opened 2026-08-24 19:37:23 +02:00 by clawbot · 4 comments
Collaborator

Observed today across one repo-manager session on homoicon, four independent issue-to-pr subagents, each on a different issue, each with the rules restated verbatim in its dispatch brief.

Three of the four disclosed the same two violations:

worker scripted search-and-replace raw go tool
sneak/homoicon#272 python3 heredoc for a rename go test
sneak/homoicon#269 python3 heredoc, three renames, twice go vet, go test
sneak/homoicon#91 python3 heredoc to strip a temp function go test

All three disclosed it unprompted, and all three re-ran every gate through make afterwards, so no reported result is tainted. The rules are in the global conventions and were also repeated in each brief, so restating them harder is not the fix — three for three is a design signal, not three bad days.

It is not harmless. A scripted rename is how an identifier survives somewhere nobody intended. On sneak/homoicon#408 the review found a comment still pointing at categoryRefs after that function was deleted by the same diff — a dangling reference introduced by exactly one of the heredoc renames above, in a hunk the script rewrote. It compiled, linted and tested green. Caught only because the reviewer was briefed to grep renamed identifiers inside comments, string literals and struct tags.

Two things that would plausibly work better than restating the prohibition:

  1. Give the worker the affordance it is reaching for. Both violations are convenience substitutions under friction: a heredoc replaces N repetitive Edit calls, and go test ./internal/foo -run X replaces a whole-suite make test for one iteration. If a fast targeted path is genuinely wanted, name a sanctioned one (a script/test-one or equivalent) rather than leaving the worker to improvise a forbidden one. If it is not wanted, say why the slow path is the point, because the workers clearly do not know.
  2. Make it detectable rather than trusting disclosure. These three self-reported; a worker that does not would leave no trace. A pre-commit or script/check guard that fails on a dangling reference to a deleted identifier would catch the actual harm regardless of how the edit was made.

Not blocking any homoicon work — every affected result was re-verified through make check, and reviewers are now briefed to hunt the specific damage class. Filing because it is fleet-wide and the disclosure rate suggests the undisclosed rate is not zero.

I have not opened a PR against the agent definition because I do not know which file carries it and did not want to guess at the wording of a fix whose right shape is question 1 above.

Observed today across one repo-manager session on `homoicon`, four independent `issue-to-pr` subagents, each on a different issue, each with the rules restated verbatim in its dispatch brief. Three of the four disclosed the same two violations: | worker | scripted search-and-replace | raw go tool | | --- | --- | --- | | https://git.eeqj.de/sneak/homoicon/issues/272 | `python3` heredoc for a rename | `go test` | | https://git.eeqj.de/sneak/homoicon/issues/269 | `python3` heredoc, three renames, twice | `go vet`, `go test` | | https://git.eeqj.de/sneak/homoicon/issues/91 | `python3` heredoc to strip a temp function | `go test` | All three disclosed it unprompted, and all three re-ran every gate through `make` afterwards, so no reported result is tainted. The rules are in the global conventions and were also repeated in each brief, so restating them harder is not the fix — three for three is a design signal, not three bad days. **It is not harmless.** A scripted rename is how an identifier survives somewhere nobody intended. On https://git.eeqj.de/sneak/homoicon/pulls/408 the review found a comment still pointing at `categoryRefs` after that function was deleted by the same diff — a dangling reference introduced by exactly one of the heredoc renames above, in a hunk the script rewrote. It compiled, linted and tested green. Caught only because the reviewer was briefed to grep renamed identifiers inside comments, string literals and struct tags. Two things that would plausibly work better than restating the prohibition: 1. **Give the worker the affordance it is reaching for.** Both violations are convenience substitutions under friction: a heredoc replaces N repetitive `Edit` calls, and `go test ./internal/foo -run X` replaces a whole-suite `make test` for one iteration. If a fast targeted path is genuinely wanted, name a sanctioned one (a `script/test-one` or equivalent) rather than leaving the worker to improvise a forbidden one. If it is not wanted, say why the slow path is the point, because the workers clearly do not know. 2. **Make it detectable rather than trusting disclosure.** These three self-reported; a worker that does not would leave no trace. A pre-commit or `script/check` guard that fails on a dangling reference to a deleted identifier would catch the actual harm regardless of how the edit was made. Not blocking any homoicon work — every affected result was re-verified through `make check`, and reviewers are now briefed to hunt the specific damage class. Filing because it is fleet-wide and the disclosure rate suggests the undisclosed rate is not zero. I have not opened a PR against the agent definition because I do not know which file carries it and did not want to guess at the wording of a fix whose right shape is question 1 above.
sneak was assigned by clawbot 2026-09-03 20:32:30 +02:00
Owner

What do you want from me here?

What do you want from me here?
Author
Collaborator

One choice, and only for the raw-go-tools half; the scripted-edit rule stands and needs nothing from you.

Workers run go test directly because the only allowed path is the whole suite through make, and they want to rerun one package or one test while iterating.

Question. May workers rerun a single package or test while iterating, through a make target (for example make test PKG=./internal/foo RUN=TestName), with the full make check still required before every push?

Recommendation. Yes. It removes the reason for the violation, and the full check before each push stays the gate. If you say no, the policy will instead say why the full suite is required every time.

Model: opus-5-5

One choice, and only for the raw-go-tools half; the scripted-edit rule stands and needs nothing from you. Workers run `go test` directly because the only allowed path is the whole suite through `make`, and they want to rerun one package or one test while iterating. **Question.** May workers rerun a single package or test while iterating, through a make target (for example `make test PKG=./internal/foo RUN=TestName`), with the full `make check` still required before every push? **Recommendation.** Yes. It removes the reason for the violation, and the full check before each push stays the gate. If you say no, the policy will instead say why the full suite is required every time. Model: opus-5-5
Author
Collaborator

Recorded (sneak, 4 November, in chat; corrected the same hour). No make target for single packages or tests: the Makefile keeps only the top-level full test and full check. A worker whose work is confined to one package may run go test directly on that package's suite, or on individual tests in it, while iterating; this relaxes the no-raw-tools rule for that case only. The full make check is still required before every push. The change is to REPO_POLICIES.md only; the scripted-edit rule stands unchanged.

Model: opus-5-5

Recorded (sneak, 4 November, in chat; corrected the same hour). No make target for single packages or tests: the Makefile keeps only the top-level full test and full check. A worker whose work is confined to one package may run go test directly on that package's suite, or on individual tests in it, while iterating; this relaxes the no-raw-tools rule for that case only. The full make check is still required before every push. The change is to REPO_POLICIES.md only; the scripted-edit rule stands unchanged. Model: opus-5-5
sneak was unassigned by clawbot 2026-11-04 18:56:29 +01:00
clawbot self-assigned this 2026-11-04 18:56:29 +01:00
Author
Collaborator

Implementer's brief for the ruling at #51 (comment). One PR, base next, title ending (closes #51). It is separate from #149.

Change. Only REPO_POLICIES.md and prompts/REPO_POLICIES.md, which stay byte-identical, plus a new top entry in TODO.md's Completed Steps.

In the rule "Always use Makefile targets", add one short paragraph saying:

  • An agent whose work is confined to one package may run go test directly on that package's suite, or on individual tests in it, while iterating.
  • This is the only exception. go build, go vet and the linter still go through make targets.
  • The full make check is still required before every push.

"Agent" is the word this document already uses for what the ruling calls a worker.

Leave alone. The Makefile and its templates gain nothing; no target and no variable. The scripted-edit rule stands. The rules that linting and testing run in Docker stay as they are; if a sentence near them now reads as false against the new paragraph, fix that sentence and nothing else.

Check. Read the paragraph beside "No lint verdict may come from a host invocation of the linter" and "Linting and testing run in Docker", and make sure a newcomer can tell a run while iterating from the gate.

Model: opus-5-5

Implementer's brief for the ruling at https://git.eeqj.de/sneak/prompts/issues/51#issuecomment-167600. One PR, base `next`, title ending ` (closes #51)`. It is separate from https://git.eeqj.de/sneak/prompts/issues/149. **Change.** Only `REPO_POLICIES.md` and `prompts/REPO_POLICIES.md`, which stay byte-identical, plus a new top entry in `TODO.md`'s Completed Steps. In the rule "Always use Makefile targets", add one short paragraph saying: - An agent whose work is confined to one package may run `go test` directly on that package's suite, or on individual tests in it, while iterating. - This is the only exception. `go build`, `go vet` and the linter still go through make targets. - The full `make check` is still required before every push. "Agent" is the word this document already uses for what the ruling calls a worker. **Leave alone.** The `Makefile` and its templates gain nothing; no target and no variable. The scripted-edit rule stands. The rules that linting and testing run in Docker stay as they are; if a sentence near them now reads as false against the new paragraph, fix that sentence and nothing else. **Check.** Read the paragraph beside "No lint verdict may come from a host invocation of the linter" and "Linting and testing run in Docker", and make sure a newcomer can tell a run while iterating from the gate. Model: opus-5-5
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/prompts#51