Test recipe reruns a hung suite, so a hang costs the 90s backstop twice and can exceed the 5 minute build limit #61

Closed
opened 2026-09-23 01:43:41 +02:00 by clawbot · 3 comments
Collaborator

Proposed policy change, moved here from sneak/vaultik#101 per your ruling that policy problems belong upstream.

The 30s timeout that issue objected to is already gone: REPO_POLICIES.md on main sets a 60 second cap and a 90 second -timeout backstop (ruling at #41 (comment)). One conflict from that issue is still open.

The conflict. The canonical test recipe (REPO_POLICIES.md:183-185, :283-285) reruns the suite with -v when it fails. When a test hangs, the first run waits the full 90s and the rerun waits another 90s. That is 180s of test time before the build even counts compiling and linting, and :315 says Docker builds must finish in under 5 minutes. vaultik measured a hang build at about 6.5 minutes with a 120s timeout. At 90s it is still close to or over the limit.

Options.

  1. Skip the verbose rerun when the first run timed out, so a hang uses the backstop only once.
  2. Say that the 5 minute build limit does not apply to a build that fails on a hang.
  3. Leave both rules as they are and accept that a hung build breaks the time limit.

Recommendation: option 1. A timeout already prints every goroutine's stack, so a verbose rerun adds nothing, and the build time limit stays meaningful.

Definition of done. The recipe and the build-time text on main agree, and consuming repos pick it up at their next re-vendor.

Model: opus-5-5

Proposed policy change, moved here from https://git.eeqj.de/sneak/vaultik/issues/101 per your ruling that policy problems belong upstream. The 30s timeout that issue objected to is already gone: `REPO_POLICIES.md` on `main` sets a 60 second cap and a 90 second `-timeout` backstop (ruling at https://git.eeqj.de/sneak/prompts/issues/41#issuecomment-53166). One conflict from that issue is still open. **The conflict.** The canonical test recipe (`REPO_POLICIES.md:183-185`, `:283-285`) reruns the suite with `-v` when it fails. When a test hangs, the first run waits the full 90s and the rerun waits another 90s. That is 180s of test time before the build even counts compiling and linting, and `:315` says Docker builds must finish in under 5 minutes. vaultik measured a hang build at about 6.5 minutes with a 120s timeout. At 90s it is still close to or over the limit. **Options.** 1. Skip the verbose rerun when the first run timed out, so a hang uses the backstop only once. 2. Say that the 5 minute build limit does not apply to a build that fails on a hang. 3. Leave both rules as they are and accept that a hung build breaks the time limit. **Recommendation: option 1.** A timeout already prints every goroutine's stack, so a verbose rerun adds nothing, and the build time limit stays meaningful. **Definition of done.** The recipe and the build-time text on `main` agree, and consuming repos pick it up at their next re-vendor. Model: opus-5-5
sneak was assigned by clawbot 2026-09-23 01:43:41 +02:00
Owner

1 approved

1 approved
Author
Collaborator

Taken, per sneak's approval of option 1. Implementer's brief:

  • Go recipe. In prompts/REPO_POLICIES.md, change the test phase of the canonical Go Dockerfile and the Go make test example the same way. The first run still prints its output. On a failure, look in that output for what Go prints for a hang: panic: test timed out (the test's own timer) or *** Test killed (go test stopping a test that ignored it). If either is there, exit 1 with no rerun, because the stacks are already printed. Any other failure reruns with -v as now. POSIX sh, no pipefail.
  • General pattern. The <test-command> example and the paragraph above it say a timed-out first run gets no rerun.
  • Python. The Dockerfile example's timeout 90 sh -c already gives both runs one 90 seconds, so its comment says that. The Python make test example has no timeout at all, so it takes the same wrapper.
  • Build-time text. The 60 and 90 second rule and the 5-minute rule say a hung package uses the backstop once. -timeout applies to each package, so do not promise more than that.
  • Try it. Plant a hanging test and a plain failing test in a scratch Go module, and run the recipe in the Debian Go image. The hang fails after one backstop with no rerun, and the plain failure still reruns. A shorter -timeout is fine for the trial.
  • last_modified and TODO.md as usual. No other rule changes.

Model: opus-5-5

Taken, per sneak's approval of option 1. Implementer's brief: - **Go recipe.** In `prompts/REPO_POLICIES.md`, change the `test` phase of the canonical Go `Dockerfile` and the Go `make test` example the same way. The first run still prints its output. On a failure, look in that output for what Go prints for a hang: `panic: test timed out` (the test's own timer) or `*** Test killed` (`go test` stopping a test that ignored it). If either is there, exit 1 with no rerun, because the stacks are already printed. Any other failure reruns with `-v` as now. POSIX sh, no `pipefail`. - **General pattern.** The `<test-command>` example and the paragraph above it say a timed-out first run gets no rerun. - **Python.** The `Dockerfile` example's `timeout 90 sh -c` already gives both runs one 90 seconds, so its comment says that. The Python `make test` example has no timeout at all, so it takes the same wrapper. - **Build-time text.** The 60 and 90 second rule and the 5-minute rule say a hung package uses the backstop once. `-timeout` applies to each package, so do not promise more than that. - **Try it.** Plant a hanging test and a plain failing test in a scratch Go module, and run the recipe in the Debian Go image. The hang fails after one backstop with no rerun, and the plain failure still reruns. A shorter `-timeout` is fine for the trial. - `last_modified` and `TODO.md` as usual. No other rule changes. Model: opus-5-5
sneak was unassigned by clawbot 2026-11-03 21:27:38 +01:00
clawbot self-assigned this 2026-11-03 21:27:38 +01:00
Author
Collaborator

Built in #143.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/prompts/pulls/143. Model: opus-5-5
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/prompts#61