sneak/prompts
1
0
Fork 0
Code Issues 18 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Raise org-wide make test cap to 60s, backstop timeout to 90s #42

Merged
sneak merged 1 commits from org-wide-60s-test-cap into main 2026-08-10 16:13:31 +02:00
Conversation 0 Commits 1 Files Changed 3 +14 -7
clawbot commented 2026-08-10 14:54:28 +02:00
Collaborator
Copy Link

The question this answers

Is the 60-second test-time cap the new org-wide ceiling, or an approved
dnswatcher-only divergence?

This PR implements org-wide.

The ruling

On sneak/dnswatcher#93 (comment) (2026-08-09),
verbatim:

> make the cap 60s in both and never use mocking, always use live resolvers and
> assume the build and run environments have full unmodified unrestricted
> internet access. it is ok if they fail due to a bad build environment that
> alters dns packets.

And on #41 (comment), disambiguating
the scope:

> org wide. the hard cap is 60 for ci/green, but over 20s should be filed as an
> improvement bug.

That second comment landed after this work was started, and it confirms the
option implemented here. The two-tier shape it describes (60s hard, 20s target,
overage filed as a bug) is encoded in the policy text.

What changed, and the number chosen

The backstop moves from 30s to 90s.

The old pairing was incoherent under the new cap: a 60-second ceiling with a
30-second -timeout means the timeout kills the suite long before the ceiling
is reached, so the ceiling would never be the thing that fails. The backstop has
to sit above the cap, where it does its actual job of catching a hung test
rather than a merely slow one. 90s preserves the 1.5x backstop-to-cap ratio
the old 20s/30s pair already had, so the relationship between the two
numbers is unchanged and only the scale moves.

Every place a number changed:

File What
prompts/REPO_POLICIES.md Prose ceiling: 20 seconds to 60 seconds, plus the new 20s target / improvement-bug tier
prompts/REPO_POLICIES.md Backstop prose: 30-second timeout to 90-second timeout, with the rationale for why it exceeds the cap
prompts/REPO_POLICIES.md Go example Makefile snippet, first run: go test -timeout 30s to -timeout 90s
prompts/REPO_POLICIES.md Go example Makefile snippet, verbose rerun: go test -timeout 30s to -timeout 90s
prompts/EXISTING_REPO_CHECKLIST.md make test has a 30-second timeout, to 90-second timeout plus the 60s hard cap and the 20s filing rule
prompts/NEW_REPO_CHECKLIST.md script/test / make test entrypoint line: 30-second timeout to 90-second timeout, 60-second hard cap on wall time

I swept the whole repo for 20 second, 30-second, 20s, 30s, timeout 20,
timeout 30, and under 20/under 30. After this change the only remaining
occurrences of 20 in a test-timing context are the two intentional references
to the new 20-second target. The other timeout hits in the repo are unrelated
(.golangci.yml lint timeout, HTTP server ReadTimeout/WriteTimeout examples,
middleware.Timeout) and were left alone.

One deliberate non-change: the Python example Makefile snippet in
prompts/REPO_POLICIES.md carries no timeout flag today and still carries none.
pytest has no built-in timeout, so adding one would mean mandating the
pytest-timeout plugin org-wide, which is a new dependency requirement rather
than a renumbering, and outside what was ruled on. It is a pre-existing gap
between the prose and that snippet, not one this PR introduces. Happy to file it
separately or add --timeout=90 here if you want it in scope.

The alternative that was not implemented

Keep canonical at 20s and let sneak/dnswatcher carry a documented per-repo
divergence. That was the recommendation originally written up in
#41, on the reasoning that the 20s
ceiling is doing real work in repos with fast deterministic suites and only
dnswatcher needs the headroom.

Org-wide was chosen instead for two reasons. First, the pressure is not specific
to DNS: any repo whose tests exercise real infrastructure over the network
inherits the same variance, and there is no principled line that admits
dnswatcher and excludes the next such repo. Second, and more decisively,
REPO_POLICIES.md is a vendored file. A sanctioned per-repo divergence in a
vendored file is indistinguishable, on inspection, from a stale vendored copy:
the next re-vendoring silently reverts the divergence, and nobody reading a
consuming repo can tell whether the number they are looking at is an intentional
exception or drift. That is the bidirectional-drift problem already tracked in
#31. The two-tier cap gets the same
outcome without the drift, since a fast repo that regresses from 4s to 45s still
generates an improvement bug.

Status

This PR was opened speculatively, ahead of a decision, on the standing "open it
rather than wait" instruction. The scope question has since been answered
org-wide in the issue, but the specific backstop value of 90s and the two-tier
wording are still my proposals rather than anything ruled on, so closing this or
sending it back for a different number is a perfectly fine outcome.

make check passes; make fmt was run and the result is included (it was a
no-op, the edits were already prettier-conformant).

sneak/dnswatcher is landing the matching 60s edit to its vendored copy in
parallel, and will match whichever way this is decided.

## The question this answers Is the 60-second test-time cap the new **org-wide** ceiling, or an approved **dnswatcher-only** divergence? - Question: https://git.eeqj.de/sneak/prompts/issues/41 - Origin: https://git.eeqj.de/sneak/dnswatcher/issues/93 This PR implements **org-wide**. ## The ruling On https://git.eeqj.de/sneak/dnswatcher/issues/93#issuecomment-50438 (2026-08-09), verbatim: > make the cap 60s in both and never use mocking, always use live resolvers and > assume the build and run environments have full unmodified unrestricted > internet access. it is ok if they fail due to a bad build environment that > alters dns packets. And on https://git.eeqj.de/sneak/prompts/issues/41#issuecomment-53166, disambiguating the scope: > org wide. the hard cap is 60 for ci/green, but over 20s should be filed as an > improvement bug. That second comment landed after this work was started, and it confirms the option implemented here. The two-tier shape it describes (60s hard, 20s target, overage filed as a bug) is encoded in the policy text. ## What changed, and the number chosen The backstop moves from `30s` to **`90s`**. The old pairing was incoherent under the new cap: a 60-second ceiling with a 30-second `-timeout` means the timeout kills the suite long before the ceiling is reached, so the ceiling would never be the thing that fails. The backstop has to sit above the cap, where it does its actual job of catching a hung test rather than a merely slow one. `90s` preserves the 1.5x backstop-to-cap ratio the old `20s`/`30s` pair already had, so the relationship between the two numbers is unchanged and only the scale moves. Every place a number changed: | File | What | | --- | --- | | `prompts/REPO_POLICIES.md` | Prose ceiling: `20 seconds` to `60 seconds`, plus the new 20s target / improvement-bug tier | | `prompts/REPO_POLICIES.md` | Backstop prose: `30-second timeout` to `90-second timeout`, with the rationale for why it exceeds the cap | | `prompts/REPO_POLICIES.md` | Go example Makefile snippet, first run: `go test -timeout 30s` to `-timeout 90s` | | `prompts/REPO_POLICIES.md` | Go example Makefile snippet, verbose rerun: `go test -timeout 30s` to `-timeout 90s` | | `prompts/EXISTING_REPO_CHECKLIST.md` | `make test` has a `30-second` timeout, to `90-second` timeout plus the 60s hard cap and the 20s filing rule | | `prompts/NEW_REPO_CHECKLIST.md` | `script/test` / `make test` entrypoint line: `30-second timeout` to `90-second timeout, 60-second hard cap on wall time` | I swept the whole repo for `20 second`, `30-second`, `20s`, `30s`, `timeout 20`, `timeout 30`, and `under 20`/`under 30`. After this change the only remaining occurrences of `20` in a test-timing context are the two intentional references to the new 20-second target. The other `timeout` hits in the repo are unrelated (`.golangci.yml` lint timeout, HTTP server `ReadTimeout`/`WriteTimeout` examples, `middleware.Timeout`) and were left alone. One deliberate non-change: the Python example Makefile snippet in `prompts/REPO_POLICIES.md` carries no timeout flag today and still carries none. `pytest` has no built-in timeout, so adding one would mean mandating the `pytest-timeout` plugin org-wide, which is a new dependency requirement rather than a renumbering, and outside what was ruled on. It is a pre-existing gap between the prose and that snippet, not one this PR introduces. Happy to file it separately or add `--timeout=90` here if you want it in scope. ## The alternative that was not implemented **Keep canonical at 20s and let `sneak/dnswatcher` carry a documented per-repo divergence.** That was the recommendation originally written up in https://git.eeqj.de/sneak/prompts/issues/41, on the reasoning that the 20s ceiling is doing real work in repos with fast deterministic suites and only dnswatcher needs the headroom. Org-wide was chosen instead for two reasons. First, the pressure is not specific to DNS: any repo whose tests exercise real infrastructure over the network inherits the same variance, and there is no principled line that admits dnswatcher and excludes the next such repo. Second, and more decisively, `REPO_POLICIES.md` is a vendored file. A sanctioned per-repo divergence in a vendored file is indistinguishable, on inspection, from a stale vendored copy: the next re-vendoring silently reverts the divergence, and nobody reading a consuming repo can tell whether the number they are looking at is an intentional exception or drift. That is the bidirectional-drift problem already tracked in https://git.eeqj.de/sneak/prompts/issues/31. The two-tier cap gets the same outcome without the drift, since a fast repo that regresses from 4s to 45s still generates an improvement bug. ## Status This PR was opened speculatively, ahead of a decision, on the standing "open it rather than wait" instruction. The scope question has since been answered org-wide in the issue, but the specific backstop value of `90s` and the two-tier wording are still my proposals rather than anything ruled on, so closing this or sending it back for a different number is a perfectly fine outcome. `make check` passes; `make fmt` was run and the result is included (it was a no-op, the edits were already prettier-conformant). `sneak/dnswatcher` is landing the matching 60s edit to its vendored copy in parallel, and will match whichever way this is decided.
clawbot added 1 commit 2026-08-10 14:54:28 +02:00
Raise org-wide make test cap to 60s, backstop timeout to 90s
All checks were successful
check / check (push) Successful in 12s
Details
52b51929dd 
Per the ruling on prompts issue 41: the 60-second cap is org-wide, not a
per-repo divergence. 60 seconds is the hard cap for CI green; anything
over 20 seconds is still green but must be filed as an improvement bug.

The old pairing of a 20-second cap with a 30-second `-timeout` stays
coherent only if the backstop exceeds the cap, so the timeout moves to
90s (the same 1.5x ratio the old numbers had). Applied to the prose
ceiling in `REPO_POLICIES.md`, the Go example Makefile snippet, and the
`make test` lines in both repo checklists.

#41
sneak/dnswatcher#93
sneak merged commit a8686891ba into main 2026-08-10 16:13:31 +02:00
sneak deleted branch org-wide-60s-test-cap 2026-08-10 16:13:31 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
merge-ready
PR approved and ready to merge
 needs-review
PR needs code review
 needs-rework
PR needs changes based on review
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
clawbot sneak
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/prompts#42
Delete Branch "org-wide-60s-test-cap"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?