States once, under the rerun pattern, that a timed-out first run gets no rerun and a hung package uses the backstop once, which three rules each said; splits the timeout rule from the 60-second rule and trims wording; names the examples that carry the command, and that a Makefile writes each $ as $$, since the Go and Python recipes go in their sections' commits; every limit stays. Model: opus-5-5
This commit is contained in:
1 parent
93de4f1758
commit
2f9f3dd622
1 file changed
+26
-25
+26
-25
@@ -167,25 +167,25 @@ style conventions are in separate documents:
|
|||||||
- in `.golangci.yml`: the entries the repo added to the `deny` list of the
|
- in `.golangci.yml`: the entries the repo added to the `deny` list of the
|
||||||
`test-support` depguard rule
|
`test-support` depguard rule
|
||||||
|
|
||||||
|
# Tests and checks
|
||||||
|
|
||||||
- All repos with software must have tests that run via the platform-standard
|
- All repos with software must have tests that run via the platform-standard
|
||||||
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
|
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). There is no
|
||||||
tests exist yet, add the most minimal test possible: importing the module
|
excuse for `make test` to be a no-op. If no meaningful tests exist yet, add
|
||||||
under test to verify it compiles/parses, for example. There is no excuse for
|
the most minimal one: importing the module under test to verify it
|
||||||
`make test` to be a no-op.
|
compiles/parses, for example.
|
||||||
|
|
||||||
- `make test` must complete in under 60 seconds. That is the hard cap, and a
|
- `make test` must complete in under 60 seconds. That is the hard cap, and a
|
||||||
suite that exceeds it fails. Under 20 seconds is the target. A suite between
|
suite that exceeds it fails. Under 20 seconds is the target. A suite between
|
||||||
20 and 60 seconds is still green, but the overage must be filed as an
|
20 and 60 seconds is still green, but the overage must be filed as an
|
||||||
improvement bug against that repo. Add a 90-second timeout to the test
|
improvement bug against that repo.
|
||||||
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
|
|
||||||
hard cap so that it catches a genuinely hung test rather than a merely slow
|
- Add a 90-second timeout to the test invocation (`go test -timeout 90s`). This
|
||||||
one. A hung package uses the backstop once, because a first run that timed out
|
backstop sits above the hard cap on purpose: it catches a hung test, not a
|
||||||
gets no rerun. `-timeout` applies to each package, so several hung packages
|
merely slow one. `-timeout` applies to each package, so several hung packages
|
||||||
can each use it.
|
can each use it.
|
||||||
|
|
||||||
- Docker builds must complete in under 5 minutes. A hung package uses the
|
- Docker builds must complete in under 5 minutes.
|
||||||
90-second test backstop once, because a first run that timed out gets no
|
|
||||||
rerun.
|
|
||||||
|
|
||||||
- `make check` must not modify any files in the repo. Tests may use temporary
|
- `make check` must not modify any files in the repo. Tests may use temporary
|
||||||
directories.
|
directories.
|
||||||
@@ -193,14 +193,20 @@ style conventions are in separate documents:
|
|||||||
- `main` must always pass `make check`, no exceptions.
|
- `main` must always pass `make check`, no exceptions.
|
||||||
|
|
||||||
- **The test command should use the conditional verbose rerun pattern.** Run
|
- **The test command should use the conditional verbose rerun pattern.** Run
|
||||||
tests without `-v` (verbose) first. If tests fail, automatically rerun with
|
tests without `-v` (verbose) first. If they fail, rerun automatically with
|
||||||
`-v` to show full output. This keeps CI logs and `docker build` output clean
|
`-v`. Success then leaves CI logs and `docker build` output clean, with just
|
||||||
on success, with just package/suite summaries. On failure it provides full
|
package/suite summaries. Failure shows every test case and every assertion.
|
||||||
diagnostic detail: every test case, every assertion. A first run that timed
|
- The rerun always ends in `exit 1`. The first run already proved the tests
|
||||||
out gets no rerun, because the rerun would only wait for the same hang. The
|
are broken, so the build must not pass even if a flaky test succeeds on
|
||||||
command lives in the `test` phase of the `Dockerfile`, since `script/test`
|
the second attempt. The rerun exists solely for diagnostic output.
|
||||||
builds that phase. The Makefile form below is the same pattern for any
|
- A first run that timed out gets no rerun, which would only wait for the
|
||||||
repo-local invocation:
|
same hang. A hung package therefore uses the 90-second backstop once.
|
||||||
|
|
||||||
|
The command lives in the `test` phase of the `Dockerfile`, since
|
||||||
|
`script/test` builds that phase. `examples/go.Dockerfile` and
|
||||||
|
`examples/python-test-phase.Dockerfile` in the `prompts` repo carry it. The
|
||||||
|
Makefile form below is the same pattern for any repo-local invocation. In a
|
||||||
|
Makefile, each `$` of those two commands is written `$$`.
|
||||||
|
|
||||||
```makefile
|
```makefile
|
||||||
test:
|
test:
|
||||||
@@ -209,11 +215,6 @@ style conventions are in separate documents:
|
|||||||
<test-command-with-v>; exit 1; }
|
<test-command-with-v>; exit 1; }
|
||||||
```
|
```
|
||||||
|
|
||||||
The `exit 1` ensures the target always fails after a rerun. The first run
|
|
||||||
already proved the tests are broken, so the build must not pass even if a
|
|
||||||
flaky test happens to succeed on the second attempt. The rerun exists solely
|
|
||||||
for diagnostic output.
|
|
||||||
|
|
||||||
`script/bootstrap` installs all dependencies idempotently and assumes
|
`script/bootstrap` installs all dependencies idempotently and assumes
|
||||||
nothing is present. Base tools, node and yarn come from nix, apt, brew, or
|
nothing is present. Base tools, node and yarn come from nix, apt, brew, or
|
||||||
apk, detected in that order; apt runs noninteractive. Each one is installed
|
apk, detected in that order; apt runs noninteractive. Each one is installed
|
||||||
|
|||||||
Reference in new issue
Block a user