Skip the verbose test rerun after a timeout (closes #61)
check / check (push) Canceled after 0s

A hung test used the 90-second backstop twice, because the verbose
rerun waited for the same hang. The Go recipe now looks in the first
run's output for the lines Go prints for a hang, and exits 1 with no
rerun. A failure with neither line in its output still reruns with -v.

The Python make test example takes the same timeout wrapper as its
Dockerfile example. The 60 and 90 second rule and the 5-minute rule say
a hung package uses the backstop once. The Go item in both repo
checklists now asks for this.

Model: opus-5-5
This commit was merged in pull request #143.
This commit is contained in:
2026-11-03 23:08:23 +01:00
parent 430964ceac
commit 229ea67434
4 changed files with 74 additions and 22 deletions
+10
View File
@@ -21,6 +21,16 @@ fmt-check, and commit.
# Completed Steps # Completed Steps
- 2026-11-03: A first test run that timed out now gets no verbose rerun
(https://git.eeqj.de/sneak/prompts/issues/61), on sneak's approval at
https://git.eeqj.de/sneak/prompts/issues/61#issuecomment-165195. The Go recipe
in `REPO_POLICIES.md` looks in the first run's output for
`panic: test timed out` or `*** Test killed`, and exits 1 when either is
there. The Python `make test` example takes the same `timeout 90 sh -c`
wrapper as its `Dockerfile` example. The 60 and 90 second rule and the
5-minute rule now say a hung package uses the backstop once. The Go item in
both repo checklists now asks for this, so a repository with the old recipe
fails it.
- 2026-10-27: Added `prompts/API_POLICIES.md` - 2026-10-27: Added `prompts/API_POLICIES.md`
(https://git.eeqj.de/sneak/prompts/issues/137). It holds sneak's rule from (https://git.eeqj.de/sneak/prompts/issues/137). It holds sneak's rule from
https://git.eeqj.de/sneak/routewatch/issues/199 that a value in an API https://git.eeqj.de/sneak/routewatch/issues/199 that a value in an API
+4 -3
View File
@@ -1,6 +1,6 @@
--- ---
title: Existing Repo Checklist title: Existing Repo Checklist
last_modified: 2026-10-26 last_modified: 2026-11-03
--- ---
Use this checklist when beginning work in a repo that may not yet conform to our Use this checklist when beginning work in a repo that may not yet conform to our
@@ -70,8 +70,9 @@ with your task.
(`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run` (`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run`
also mounts golangci-lint's also mounts golangci-lint's
(`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step (`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step
mounts anything else; both `go test` runs carry `-count=1`, as in the mounts anything else; both `go test` runs carry `-count=1`; a first run
canonical Go `Dockerfile` in `REPO_POLICIES.md` that timed out gets no rerun, as in the canonical Go `Dockerfile` in
`REPO_POLICIES.md`
- [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does - [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does
it above `ARG CACHEBUST`, with `COPY script/ script/`, it above `ARG CACHEBUST`, with `COPY script/ script/`,
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When
+4 -3
View File
@@ -1,6 +1,6 @@
--- ---
title: New Repo Checklist title: New Repo Checklist
last_modified: 2026-10-26 last_modified: 2026-11-03
--- ---
Use this checklist when creating a new repository from scratch. Follow the steps Use this checklist when creating a new repository from scratch. Follow the steps
@@ -129,8 +129,9 @@ Template files can be fetched from:
(`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run` (`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run`
also mounts golangci-lint's also mounts golangci-lint's
(`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step (`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step
mounts anything else; both `go test` runs carry `-count=1`, as in the mounts anything else; both `go test` runs carry `-count=1`; a first run
canonical Go `Dockerfile` in `REPO_POLICIES.md` that timed out gets no rerun, as in the canonical Go `Dockerfile` in
`REPO_POLICIES.md`
- Python: a `test` phase that installs its packages from `Pipfile.lock` does - Python: a `test` phase that installs its packages from `Pipfile.lock` does
it above `ARG CACHEBUST`, with `COPY script/ script/`, it above `ARG CACHEBUST`, with `COPY script/ script/`,
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When
+56 -16
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-10-27 last_modified: 2026-11-03
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -233,6 +233,8 @@ style conventions are in separate documents:
# Test phase. -race needs cgo and so a C compiler, which the Debian Go # Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not. -count=1 because the mounted # image ships and the alpine one does not. -count=1 because the mounted
# cache also holds test results, which go test would otherwise replay. # cache also holds test results, which go test would otherwise replay.
# A first run that timed out gets no rerun, because a rerun would only
# wait for the same hang.
# golang:1.x, YYYY-MM-DD # golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test FROM golang@sha256:... AS test
WORKDIR /src WORKDIR /src
@@ -241,9 +243,16 @@ style conventions are in separate documents:
ARG CACHEBUST ARG CACHEBUST
COPY . . COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \ RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -cover ./... || \ status=0; \
{ echo "--- Rerunning with -v for details ---"; \ output=$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$?; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; } printf '%s\n' "$output"; \
test "$status" -eq 0 && exit 0; \
case "$output" in \
*"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \
esac; \
echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; \
exit 1
# Build stage. Nothing is wanted from either phase above; the copies # Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so nothing after them can # are what make BuildKit build them first, so nothing after them can
@@ -413,7 +422,8 @@ style conventions are in separate documents:
`pipenv install --dev --deploy`: `pipenv install --dev --deploy`:
```dockerfile ```dockerfile
# Test phase. The 90-second timeout covers both pytest runs. # Test phase. Both pytest runs share one 90-second timeout, so a hang
# uses it once and gets no rerun.
# python:3.x-slim, YYYY-MM-DD # python:3.x-slim, YYYY-MM-DD
FROM python@sha256:... AS test FROM python@sha256:... AS test
# make and git are base tools that script/bootstrap installs when they # make and git are base tools that script/bootstrap installs when they
@@ -484,15 +494,19 @@ style conventions are in separate documents:
improvement bug against that repo. Add a 90-second timeout to the test improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow hard cap so that it catches a genuinely hung test rather than a merely slow
one. one. A hung package uses the backstop once, because a first run that timed out
gets no rerun. `-timeout` applies to each package, so several hung packages
can each use it.
- **The test command should use the conditional verbose rerun pattern.** Run - **The test command should use the conditional verbose rerun pattern.** Run
tests without `-v` (verbose) first. If tests fail, automatically rerun with tests without `-v` (verbose) first. If tests fail, automatically rerun with
`-v` to show full output. This keeps CI logs and `docker build` output clean `-v` to show full output. This keeps CI logs and `docker build` output clean
on success (just package/suite summaries) while providing full diagnostic on success (just package/suite summaries) while providing full diagnostic
detail on failure (every test case, every assertion). The command lives in the detail on failure (every test case, every assertion). A first run that timed
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the out gets no rerun, because the rerun would only wait for the same hang. The
Makefile form below is the same pattern for any repo-local invocation: command lives in the `test` phase of the `Dockerfile`, since `script/test`
builds that phase; the Makefile form below is the same pattern for any
repo-local invocation:
```makefile ```makefile
test: test:
@@ -505,11 +519,29 @@ style conventions are in separate documents:
```makefile ```makefile
test: test:
@go test -count=1 -timeout 90s -race -cover ./... || \ @status=0; \
{ echo "--- Rerunning with -v for details ---"; \ output=$$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$$?; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; } printf '%s\n' "$$output"; \
test "$$status" -eq 0 && exit 0; \
case "$$output" in \
*"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \
esac; \
echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; \
exit 1
``` ```
The first run's output is kept so that the recipe can look in it for what Go
prints for a hang:
- `panic: test timed out` means the test's own timer fired.
- `*** Test killed` means `go test` stopped a test that ignored that timer,
about a minute later.
Go prints the goroutine stacks with either line, as long as the test can
still respond. A rerun would only wait for the same hang, so the recipe
exits 1. A failure with neither line in its output reruns with `-v`. The
kept output prints when the first run ends, not while it runs.
`-count=1` is required on both invocations: it defeats Go's test _result_ `-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite tests. It leaves the build cache alone, so it costs the runtime of the suite
@@ -528,17 +560,25 @@ style conventions are in separate documents:
```makefile ```makefile
test: test:
@python -m pytest || \ @timeout 90 sh -c 'python -m pytest || { \
{ echo "--- Rerunning with -v for details ---"; \ echo "--- Rerunning with -v for details ---"; \
python -m pytest -v; exit 1; } python -m pytest -v; exit 1; }'
``` ```
Both pytest runs share that one 90-second timeout, as in the `Dockerfile`
example above, so a hang uses it once and gets no rerun. Ctrl-C does not
stop a running `make test`, which goes on until the tests end or the 90
seconds run out. On a Mac without `timeout`, the target fails until it is
installed.
The `exit 1` ensures the target always fails after a rerun — the first run The `exit 1` ensures the target always fails after a rerun — the first run
already proved the tests are broken, so the build must not pass even if a already proved the tests are broken, so the build must not pass even if a
flaky test happens to succeed on the second attempt. The rerun exists solely flaky test happens to succeed on the second attempt. The rerun exists solely
for diagnostic output. for diagnostic output.
- Docker builds must complete in under 5 minutes. - Docker builds must complete in under 5 minutes. A hung package uses the
90-second test backstop once, because a first run that timed out gets no
rerun.
- `make check` must not modify any files in the repo. Tests may use temporary - `make check` must not modify any files in the repo. Tests may use temporary
directories. directories.