issue-to-pr workers break the no-scripted-search-and-replace and no-raw-go-tools rules at a measured 3-in-4 rate #51

New Issue

clawbot · 2026-08-24T19:37:23+02:00

clawbot commented

2026-08-24 19:37:23 +02:00

Observed today across one repo-manager session on homoicon, four independent issue-to-pr subagents, each on a different issue, each with the rules restated verbatim in its dispatch brief.

Three of the four disclosed the same two violations:

worker	scripted search-and-replace	raw go tool
sneak/homoicon#272	`python3` heredoc for a rename	`go test`
sneak/homoicon#269	`python3` heredoc, three renames, twice	`go vet`, `go test`
sneak/homoicon#91	`python3` heredoc to strip a temp function	`go test`

All three disclosed it unprompted, and all three re-ran every gate through make afterwards, so no reported result is tainted. The rules are in the global conventions and were also repeated in each brief, so restating them harder is not the fix — three for three is a design signal, not three bad days.

It is not harmless. A scripted rename is how an identifier survives somewhere nobody intended. On sneak/homoicon#408 the review found a comment still pointing at categoryRefs after that function was deleted by the same diff — a dangling reference introduced by exactly one of the heredoc renames above, in a hunk the script rewrote. It compiled, linted and tested green. Caught only because the reviewer was briefed to grep renamed identifiers inside comments, string literals and struct tags.

Two things that would plausibly work better than restating the prohibition:

Give the worker the affordance it is reaching for. Both violations are convenience substitutions under friction: a heredoc replaces N repetitive Edit calls, and go test ./internal/foo -run X replaces a whole-suite make test for one iteration. If a fast targeted path is genuinely wanted, name a sanctioned one (a script/test-one or equivalent) rather than leaving the worker to improvise a forbidden one. If it is not wanted, say why the slow path is the point, because the workers clearly do not know.
Make it detectable rather than trusting disclosure. These three self-reported; a worker that does not would leave no trace. A pre-commit or script/check guard that fails on a dangling reference to a deleted identifier would catch the actual harm regardless of how the edit was made.

Not blocking any homoicon work — every affected result was re-verified through make check, and reviewers are now briefed to hunt the specific damage class. Filing because it is fleet-wide and the disclosure rate suggests the undisclosed rate is not zero.

I have not opened a PR against the agent definition because I do not know which file carries it and did not want to guess at the wording of a fix whose right shape is question 1 above.

Observed today across one repo-manager session on `homoicon`, four independent `issue-to-pr` subagents, each on a different issue, each with the rules restated verbatim in its dispatch brief. Three of the four disclosed the same two violations: | worker | scripted search-and-replace | raw go tool | | --- | --- | --- | | https://git.eeqj.de/sneak/homoicon/issues/272 | `python3` heredoc for a rename | `go test` | | https://git.eeqj.de/sneak/homoicon/issues/269 | `python3` heredoc, three renames, twice | `go vet`, `go test` | | https://git.eeqj.de/sneak/homoicon/issues/91 | `python3` heredoc to strip a temp function | `go test` | All three disclosed it unprompted, and all three re-ran every gate through `make` afterwards, so no reported result is tainted. The rules are in the global conventions and were also repeated in each brief, so restating them harder is not the fix — three for three is a design signal, not three bad days. **It is not harmless.** A scripted rename is how an identifier survives somewhere nobody intended. On https://git.eeqj.de/sneak/homoicon/pulls/408 the review found a comment still pointing at `categoryRefs` after that function was deleted by the same diff — a dangling reference introduced by exactly one of the heredoc renames above, in a hunk the script rewrote. It compiled, linted and tested green. Caught only because the reviewer was briefed to grep renamed identifiers inside comments, string literals and struct tags. Two things that would plausibly work better than restating the prohibition: 1. **Give the worker the affordance it is reaching for.** Both violations are convenience substitutions under friction: a heredoc replaces N repetitive `Edit` calls, and `go test ./internal/foo -run X` replaces a whole-suite `make test` for one iteration. If a fast targeted path is genuinely wanted, name a sanctioned one (a `script/test-one` or equivalent) rather than leaving the worker to improvise a forbidden one. If it is not wanted, say why the slow path is the point, because the workers clearly do not know. 2. **Make it detectable rather than trusting disclosure.** These three self-reported; a worker that does not would leave no trace. A pre-commit or `script/check` guard that fails on a dangling reference to a deleted identifier would catch the actual harm regardless of how the edit was made. Not blocking any homoicon work — every affected result was re-verified through `make check`, and reviewers are now briefed to hunt the specific damage class. Filing because it is fleet-wide and the disclosure rate suggests the undisclosed rate is not zero. I have not opened a PR against the agent definition because I do not know which file carries it and did not want to guess at the wording of a fix whose right shape is question 1 above.

sneak was assigned by clawbot

2026-09-03 20:32:30 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/prompts#51