Skip the verbose test rerun after a timeout (closes #61)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
A hung test used the 90-second backstop twice, because the verbose rerun waited for the same hang. The Go recipe now looks in the first run's output for the lines Go prints for a hang, and exits 1 with no rerun. A failure with neither line in its output still reruns with -v. The Python make test example takes the same timeout wrapper as its Dockerfile example. The 60 and 90 second rule and the 5-minute rule say a hung package uses the backstop once. The Go item in both repo checklists now asks for this. Model: opus-5-5
This commit was merged in pull request #143.
This commit is contained in:
@@ -21,6 +21,16 @@ fmt-check, and commit.
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-11-03: A first test run that timed out now gets no verbose rerun
|
||||||
|
(https://git.eeqj.de/sneak/prompts/issues/61), on sneak's approval at
|
||||||
|
https://git.eeqj.de/sneak/prompts/issues/61#issuecomment-165195. The Go recipe
|
||||||
|
in `REPO_POLICIES.md` looks in the first run's output for
|
||||||
|
`panic: test timed out` or `*** Test killed`, and exits 1 when either is
|
||||||
|
there. The Python `make test` example takes the same `timeout 90 sh -c`
|
||||||
|
wrapper as its `Dockerfile` example. The 60 and 90 second rule and the
|
||||||
|
5-minute rule now say a hung package uses the backstop once. The Go item in
|
||||||
|
both repo checklists now asks for this, so a repository with the old recipe
|
||||||
|
fails it.
|
||||||
- 2026-10-27: Added `prompts/API_POLICIES.md`
|
- 2026-10-27: Added `prompts/API_POLICIES.md`
|
||||||
(https://git.eeqj.de/sneak/prompts/issues/137). It holds sneak's rule from
|
(https://git.eeqj.de/sneak/prompts/issues/137). It holds sneak's rule from
|
||||||
https://git.eeqj.de/sneak/routewatch/issues/199 that a value in an API
|
https://git.eeqj.de/sneak/routewatch/issues/199 that a value in an API
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Existing Repo Checklist
|
title: Existing Repo Checklist
|
||||||
last_modified: 2026-10-26
|
last_modified: 2026-11-03
|
||||||
---
|
---
|
||||||
|
|
||||||
Use this checklist when beginning work in a repo that may not yet conform to our
|
Use this checklist when beginning work in a repo that may not yet conform to our
|
||||||
@@ -70,8 +70,9 @@ with your task.
|
|||||||
(`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run`
|
(`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run`
|
||||||
also mounts golangci-lint's
|
also mounts golangci-lint's
|
||||||
(`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step
|
(`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step
|
||||||
mounts anything else; both `go test` runs carry `-count=1`, as in the
|
mounts anything else; both `go test` runs carry `-count=1`; a first run
|
||||||
canonical Go `Dockerfile` in `REPO_POLICIES.md`
|
that timed out gets no rerun, as in the canonical Go `Dockerfile` in
|
||||||
|
`REPO_POLICIES.md`
|
||||||
- [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does
|
- [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does
|
||||||
it above `ARG CACHEBUST`, with `COPY script/ script/`,
|
it above `ARG CACHEBUST`, with `COPY script/ script/`,
|
||||||
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When
|
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: New Repo Checklist
|
title: New Repo Checklist
|
||||||
last_modified: 2026-10-26
|
last_modified: 2026-11-03
|
||||||
---
|
---
|
||||||
|
|
||||||
Use this checklist when creating a new repository from scratch. Follow the steps
|
Use this checklist when creating a new repository from scratch. Follow the steps
|
||||||
@@ -129,8 +129,9 @@ Template files can be fetched from:
|
|||||||
(`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run`
|
(`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run`
|
||||||
also mounts golangci-lint's
|
also mounts golangci-lint's
|
||||||
(`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step
|
(`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step
|
||||||
mounts anything else; both `go test` runs carry `-count=1`, as in the
|
mounts anything else; both `go test` runs carry `-count=1`; a first run
|
||||||
canonical Go `Dockerfile` in `REPO_POLICIES.md`
|
that timed out gets no rerun, as in the canonical Go `Dockerfile` in
|
||||||
|
`REPO_POLICIES.md`
|
||||||
- Python: a `test` phase that installs its packages from `Pipfile.lock` does
|
- Python: a `test` phase that installs its packages from `Pipfile.lock` does
|
||||||
it above `ARG CACHEBUST`, with `COPY script/ script/`,
|
it above `ARG CACHEBUST`, with `COPY script/ script/`,
|
||||||
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When
|
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When
|
||||||
|
|||||||
+56
-16
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
title: Repository Policies
|
title: Repository Policies
|
||||||
last_modified: 2026-10-27
|
last_modified: 2026-11-03
|
||||||
---
|
---
|
||||||
|
|
||||||
This document covers repository structure, tooling, and workflow standards. Code
|
This document covers repository structure, tooling, and workflow standards. Code
|
||||||
@@ -233,6 +233,8 @@ style conventions are in separate documents:
|
|||||||
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
|
||||||
# image ships and the alpine one does not. -count=1 because the mounted
|
# image ships and the alpine one does not. -count=1 because the mounted
|
||||||
# cache also holds test results, which go test would otherwise replay.
|
# cache also holds test results, which go test would otherwise replay.
|
||||||
|
# A first run that timed out gets no rerun, because a rerun would only
|
||||||
|
# wait for the same hang.
|
||||||
# golang:1.x, YYYY-MM-DD
|
# golang:1.x, YYYY-MM-DD
|
||||||
FROM golang@sha256:... AS test
|
FROM golang@sha256:... AS test
|
||||||
WORKDIR /src
|
WORKDIR /src
|
||||||
@@ -241,9 +243,16 @@ style conventions are in separate documents:
|
|||||||
ARG CACHEBUST
|
ARG CACHEBUST
|
||||||
COPY . .
|
COPY . .
|
||||||
RUN --mount=type=cache,target=/root/.cache/go-build \
|
RUN --mount=type=cache,target=/root/.cache/go-build \
|
||||||
go test -count=1 -timeout 90s -race -cover ./... || \
|
status=0; \
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
output=$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$?; \
|
||||||
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
printf '%s\n' "$output"; \
|
||||||
|
test "$status" -eq 0 && exit 0; \
|
||||||
|
case "$output" in \
|
||||||
|
*"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \
|
||||||
|
esac; \
|
||||||
|
echo "--- Rerunning with -v for details ---"; \
|
||||||
|
go test -count=1 -timeout 90s -race -v ./...; \
|
||||||
|
exit 1
|
||||||
|
|
||||||
# Build stage. Nothing is wanted from either phase above; the copies
|
# Build stage. Nothing is wanted from either phase above; the copies
|
||||||
# are what make BuildKit build them first, so nothing after them can
|
# are what make BuildKit build them first, so nothing after them can
|
||||||
@@ -413,7 +422,8 @@ style conventions are in separate documents:
|
|||||||
`pipenv install --dev --deploy`:
|
`pipenv install --dev --deploy`:
|
||||||
|
|
||||||
```dockerfile
|
```dockerfile
|
||||||
# Test phase. The 90-second timeout covers both pytest runs.
|
# Test phase. Both pytest runs share one 90-second timeout, so a hang
|
||||||
|
# uses it once and gets no rerun.
|
||||||
# python:3.x-slim, YYYY-MM-DD
|
# python:3.x-slim, YYYY-MM-DD
|
||||||
FROM python@sha256:... AS test
|
FROM python@sha256:... AS test
|
||||||
# make and git are base tools that script/bootstrap installs when they
|
# make and git are base tools that script/bootstrap installs when they
|
||||||
@@ -484,15 +494,19 @@ style conventions are in separate documents:
|
|||||||
improvement bug against that repo. Add a 90-second timeout to the test
|
improvement bug against that repo. Add a 90-second timeout to the test
|
||||||
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
|
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
|
||||||
hard cap so that it catches a genuinely hung test rather than a merely slow
|
hard cap so that it catches a genuinely hung test rather than a merely slow
|
||||||
one.
|
one. A hung package uses the backstop once, because a first run that timed out
|
||||||
|
gets no rerun. `-timeout` applies to each package, so several hung packages
|
||||||
|
can each use it.
|
||||||
|
|
||||||
- **The test command should use the conditional verbose rerun pattern.** Run
|
- **The test command should use the conditional verbose rerun pattern.** Run
|
||||||
tests without `-v` (verbose) first. If tests fail, automatically rerun with
|
tests without `-v` (verbose) first. If tests fail, automatically rerun with
|
||||||
`-v` to show full output. This keeps CI logs and `docker build` output clean
|
`-v` to show full output. This keeps CI logs and `docker build` output clean
|
||||||
on success (just package/suite summaries) while providing full diagnostic
|
on success (just package/suite summaries) while providing full diagnostic
|
||||||
detail on failure (every test case, every assertion). The command lives in the
|
detail on failure (every test case, every assertion). A first run that timed
|
||||||
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
|
out gets no rerun, because the rerun would only wait for the same hang. The
|
||||||
Makefile form below is the same pattern for any repo-local invocation:
|
command lives in the `test` phase of the `Dockerfile`, since `script/test`
|
||||||
|
builds that phase; the Makefile form below is the same pattern for any
|
||||||
|
repo-local invocation:
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
test:
|
test:
|
||||||
@@ -505,11 +519,29 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
test:
|
test:
|
||||||
@go test -count=1 -timeout 90s -race -cover ./... || \
|
@status=0; \
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
output=$$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$$?; \
|
||||||
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
|
printf '%s\n' "$$output"; \
|
||||||
|
test "$$status" -eq 0 && exit 0; \
|
||||||
|
case "$$output" in \
|
||||||
|
*"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \
|
||||||
|
esac; \
|
||||||
|
echo "--- Rerunning with -v for details ---"; \
|
||||||
|
go test -count=1 -timeout 90s -race -v ./...; \
|
||||||
|
exit 1
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The first run's output is kept so that the recipe can look in it for what Go
|
||||||
|
prints for a hang:
|
||||||
|
- `panic: test timed out` means the test's own timer fired.
|
||||||
|
- `*** Test killed` means `go test` stopped a test that ignored that timer,
|
||||||
|
about a minute later.
|
||||||
|
|
||||||
|
Go prints the goroutine stacks with either line, as long as the test can
|
||||||
|
still respond. A rerun would only wait for the same hang, so the recipe
|
||||||
|
exits 1. A failure with neither line in its output reruns with `-v`. The
|
||||||
|
kept output prints when the first run ends, not while it runs.
|
||||||
|
|
||||||
`-count=1` is required on both invocations: it defeats Go's test _result_
|
`-count=1` is required on both invocations: it defeats Go's test _result_
|
||||||
cache, so neither run can report a stored pass in place of running the
|
cache, so neither run can report a stored pass in place of running the
|
||||||
tests. It leaves the build cache alone, so it costs the runtime of the suite
|
tests. It leaves the build cache alone, so it costs the runtime of the suite
|
||||||
@@ -528,17 +560,25 @@ style conventions are in separate documents:
|
|||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
test:
|
test:
|
||||||
@python -m pytest || \
|
@timeout 90 sh -c 'python -m pytest || { \
|
||||||
{ echo "--- Rerunning with -v for details ---"; \
|
echo "--- Rerunning with -v for details ---"; \
|
||||||
python -m pytest -v; exit 1; }
|
python -m pytest -v; exit 1; }'
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Both pytest runs share that one 90-second timeout, as in the `Dockerfile`
|
||||||
|
example above, so a hang uses it once and gets no rerun. Ctrl-C does not
|
||||||
|
stop a running `make test`, which goes on until the tests end or the 90
|
||||||
|
seconds run out. On a Mac without `timeout`, the target fails until it is
|
||||||
|
installed.
|
||||||
|
|
||||||
The `exit 1` ensures the target always fails after a rerun — the first run
|
The `exit 1` ensures the target always fails after a rerun — the first run
|
||||||
already proved the tests are broken, so the build must not pass even if a
|
already proved the tests are broken, so the build must not pass even if a
|
||||||
flaky test happens to succeed on the second attempt. The rerun exists solely
|
flaky test happens to succeed on the second attempt. The rerun exists solely
|
||||||
for diagnostic output.
|
for diagnostic output.
|
||||||
|
|
||||||
- Docker builds must complete in under 5 minutes.
|
- Docker builds must complete in under 5 minutes. A hung package uses the
|
||||||
|
90-second test backstop once, because a first run that timed out gets no
|
||||||
|
rerun.
|
||||||
|
|
||||||
- `make check` must not modify any files in the repo. Tests may use temporary
|
- `make check` must not modify any files in the repo. Tests may use temporary
|
||||||
directories.
|
directories.
|
||||||
|
|||||||
Reference in New Issue
Block a user