States once, under the rerun pattern, that a timed-out first run gets no rerun and a hung package uses the backstop once, which three rules each said; splits the timeout rule from the 60-second rule and trims wording; names the examples that carry the command, and that a Makefile writes each $ as $$, since the Go and Python recipes go in their sections' commits; every limit stays. Model: opus-5-5
This commit is contained in:
1 parent
93de4f1758
commit
2f9f3dd622
1 file changed
+26
-25
+26
-25
@@ -167,25 +167,25 @@ style conventions are in separate documents:
|
||||
- in `.golangci.yml`: the entries the repo added to the `deny` list of the
|
||||
`test-support` depguard rule
|
||||
|
||||
# Tests and checks
|
||||
|
||||
- All repos with software must have tests that run via the platform-standard
|
||||
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
|
||||
tests exist yet, add the most minimal test possible: importing the module
|
||||
under test to verify it compiles/parses, for example. There is no excuse for
|
||||
`make test` to be a no-op.
|
||||
test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). There is no
|
||||
excuse for `make test` to be a no-op. If no meaningful tests exist yet, add
|
||||
the most minimal one: importing the module under test to verify it
|
||||
compiles/parses, for example.
|
||||
|
||||
- `make test` must complete in under 60 seconds. That is the hard cap, and a
|
||||
suite that exceeds it fails. Under 20 seconds is the target. A suite between
|
||||
20 and 60 seconds is still green, but the overage must be filed as an
|
||||
improvement bug against that repo. Add a 90-second timeout to the test
|
||||
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
|
||||
hard cap so that it catches a genuinely hung test rather than a merely slow
|
||||
one. A hung package uses the backstop once, because a first run that timed out
|
||||
gets no rerun. `-timeout` applies to each package, so several hung packages
|
||||
improvement bug against that repo.
|
||||
|
||||
- Add a 90-second timeout to the test invocation (`go test -timeout 90s`). This
|
||||
backstop sits above the hard cap on purpose: it catches a hung test, not a
|
||||
merely slow one. `-timeout` applies to each package, so several hung packages
|
||||
can each use it.
|
||||
|
||||
- Docker builds must complete in under 5 minutes. A hung package uses the
|
||||
90-second test backstop once, because a first run that timed out gets no
|
||||
rerun.
|
||||
- Docker builds must complete in under 5 minutes.
|
||||
|
||||
- `make check` must not modify any files in the repo. Tests may use temporary
|
||||
directories.
|
||||
@@ -193,14 +193,20 @@ style conventions are in separate documents:
|
||||
- `main` must always pass `make check`, no exceptions.
|
||||
|
||||
- **The test command should use the conditional verbose rerun pattern.** Run
|
||||
tests without `-v` (verbose) first. If tests fail, automatically rerun with
|
||||
`-v` to show full output. This keeps CI logs and `docker build` output clean
|
||||
on success, with just package/suite summaries. On failure it provides full
|
||||
diagnostic detail: every test case, every assertion. A first run that timed
|
||||
out gets no rerun, because the rerun would only wait for the same hang. The
|
||||
command lives in the `test` phase of the `Dockerfile`, since `script/test`
|
||||
builds that phase. The Makefile form below is the same pattern for any
|
||||
repo-local invocation:
|
||||
tests without `-v` (verbose) first. If they fail, rerun automatically with
|
||||
`-v`. Success then leaves CI logs and `docker build` output clean, with just
|
||||
package/suite summaries. Failure shows every test case and every assertion.
|
||||
- The rerun always ends in `exit 1`. The first run already proved the tests
|
||||
are broken, so the build must not pass even if a flaky test succeeds on
|
||||
the second attempt. The rerun exists solely for diagnostic output.
|
||||
- A first run that timed out gets no rerun, which would only wait for the
|
||||
same hang. A hung package therefore uses the 90-second backstop once.
|
||||
|
||||
The command lives in the `test` phase of the `Dockerfile`, since
|
||||
`script/test` builds that phase. `examples/go.Dockerfile` and
|
||||
`examples/python-test-phase.Dockerfile` in the `prompts` repo carry it. The
|
||||
Makefile form below is the same pattern for any repo-local invocation. In a
|
||||
Makefile, each `$` of those two commands is written `$$`.
|
||||
|
||||
```makefile
|
||||
test:
|
||||
@@ -209,11 +215,6 @@ style conventions are in separate documents:
|
||||
<test-command-with-v>; exit 1; }
|
||||
```
|
||||
|
||||
The `exit 1` ensures the target always fails after a rerun. The first run
|
||||
already proved the tests are broken, so the build must not pass even if a
|
||||
flaky test happens to succeed on the second attempt. The rerun exists solely
|
||||
for diagnostic output.
|
||||
|
||||
`script/bootstrap` installs all dependencies idempotently and assumes
|
||||
nothing is present. Base tools, node and yarn come from nix, apt, brew, or
|
||||
apk, detected in that order; apt runs noninteractive. Each one is installed
|
||||
|
||||
Reference in new issue
Block a user