The test suite runs over the 20-second target in REPO_POLICIES.md #263

Closed
opened 2026-11-10 20:52:25 +01:00 by clawbot · 4 comments
Collaborator

REPO_POLICIES.md: make test must finish in under 60 seconds (hard cap); under 20 is the target, and an overage is filed as a bug against the repo. On next at bc8aa6b, make test on a fresh clone took 120 seconds on the shared host (heavily loaded at the time); an earlier run took 62. About 70 seconds pass before the first package reports, mostly compiling under -race; then the slow packages are internal/handlers (26 s), internal/ircserver (16 s), internal/service (14 s) and internal/db (10 s). The others take about a second each.

Definition of done

  • Find where the slow packages' tests spend their time (the per-test times of go test -v on one package, which a worker confined to that package may run directly) and remove the waiting: fixed sleeps, polling at fixed intervals, real timeouts such as the long poll, password hashing at full cost, and network lookups such as the reverse DNS lookup on session creation, which can wait up to three seconds. Replace them with synchronisation, or with shorter values that the test passes through options the code already has or a test-only hook in an export_test.go. Production defaults do not change.
  • Each of those four packages runs in under 5 seconds on its own.
  • No test is deleted, skipped or weakened, and no package's coverage drops.
  • The test stage of Dockerfile keeps -race, -timeout 90s, -p 4 and the verbose rerun.
  • If the compile step alone keeps make test over 60 seconds on this host, say so on this issue with what would cut it; do not change the compiler flags in this unit.

Read first: AGENTS.md, CONVENTIONS.md, REPO_POLICIES.md (the test section).

Model: opus-5-5

`REPO_POLICIES.md`: `make test` must finish in under 60 seconds (hard cap); under 20 is the target, and an overage is filed as a bug against the repo. On `next` at `bc8aa6b`, `make test` on a fresh clone took 120 seconds on the shared host (heavily loaded at the time); an earlier run took 62. About 70 seconds pass before the first package reports, mostly compiling under `-race`; then the slow packages are `internal/handlers` (26 s), `internal/ircserver` (16 s), `internal/service` (14 s) and `internal/db` (10 s). The others take about a second each. ## Definition of done - Find where the slow packages' tests spend their time (the per-test times of `go test -v` on one package, which a worker confined to that package may run directly) and remove the waiting: fixed sleeps, polling at fixed intervals, real timeouts such as the long poll, password hashing at full cost, and network lookups such as the reverse DNS lookup on session creation, which can wait up to three seconds. Replace them with synchronisation, or with shorter values that the test passes through options the code already has or a test-only hook in an `export_test.go`. Production defaults do not change. - Each of those four packages runs in under 5 seconds on its own. - No test is deleted, skipped or weakened, and no package's coverage drops. - The `test` stage of `Dockerfile` keeps `-race`, `-timeout 90s`, `-p 4` and the verbose rerun. - If the compile step alone keeps `make test` over 60 seconds on this host, say so on this issue with what would cut it; do not change the compiler flags in this unit. Read first: `AGENTS.md`, `CONVENTIONS.md`, `REPO_POLICIES.md` (the test section). Model: opus-5-5
clawbot self-assigned this 2026-11-10 20:52:25 +01:00
Author
Collaborator

Built in #267. The PR lists the changes and the items that are still unverified.

Compilation alone keeps make test over 60 seconds on this host. About 65 to 80 s pass before the first package reports. With an empty build cache, compiling the db test binary under -race took 83 s, and compiling broker, which has no SQLite, took 18 s. So most of that time is modernc.org/sqlite compiled with race instrumentation.

What would cut it: keep Go's build cache between runs in a BuildKit cache mount on the test stage's RUN (--mount=type=cache,target=/root/.cache/go-build), and add -count=1 so that every test still runs. A cache mount is not part of the image, so it keeps Go's caches out of the image just as go clean does now. Then only changed packages recompile.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/neoirc/pulls/267. The PR lists the changes and the items that are still unverified. Compilation alone keeps `make test` over 60 seconds on this host. About 65 to 80 s pass before the first package reports. With an empty build cache, compiling the `db` test binary under `-race` took 83 s, and compiling `broker`, which has no SQLite, took 18 s. So most of that time is `modernc.org/sqlite` compiled with race instrumentation. What would cut it: keep Go's build cache between runs in a BuildKit cache mount on the `test` stage's `RUN` (`--mount=type=cache,target=/root/.cache/go-build`), and add `-count=1` so that every test still runs. A cache mount is not part of the image, so it keeps Go's caches out of the image just as `go clean` does now. Then only changed packages recompile. Model: opus-5-5
Author
Collaborator

Amended definition of done, replacing the bullet "Each of those four packages runs in under 5 seconds on its own" (per-package times on this shared host swing with its load, so a stopwatch target cannot be checked):

  • No test in those four packages waits on a fixed sleep, fixed-interval polling, a timeout it is not testing, or full-cost password hashing. Time left is CPU work or a timeout under test, and the PR body names each such item in one line.
  • The hashing cost reaches the tests through a PasswordCost int field on db.Params, tagged optional:"true" so fx needs no provider for it; db.New keeps it on the Database and uses bcrypt.DefaultCost when it is zero. No package-level variable, no lint suppression, no new setting.

The other bullets stand.

Model: opus-5-5

Amended definition of done, replacing the bullet "Each of those four packages runs in under 5 seconds on its own" (per-package times on this shared host swing with its load, so a stopwatch target cannot be checked): - No test in those four packages waits on a fixed sleep, fixed-interval polling, a timeout it is not testing, or full-cost password hashing. Time left is CPU work or a timeout under test, and the PR body names each such item in one line. - The hashing cost reaches the tests through a `PasswordCost int` field on `db.Params`, tagged `optional:"true"` so `fx` needs no provider for it; `db.New` keeps it on the `Database` and uses `bcrypt.DefaultCost` when it is zero. No package-level variable, no lint suppression, no new setting. The other bullets stand. Model: opus-5-5
Owner

compilation obviously doesn't count in the testing time, don't be silly. the limit is the test suite itself.

compilation obviously doesn't count in the testing time, don't be silly. the limit is the test suite itself.
clawbot changed title from make test takes over the 60-second cap in REPO_POLICIES.md to The test suite runs over the 20-second target in REPO_POLICIES.md 2026-11-11 00:54:54 +01:00
Author
Collaborator

Per #263 (comment), compile time is outside the cap. The suite itself was under 60 seconds and over the 20-second target, which is what this issue now tracks (retitled). #267, which removes the waiting, is the fix; nothing is done about compile time.

Model: opus-5-5

Per https://git.eeqj.de/sneak/neoirc/issues/263#issuecomment-175964, compile time is outside the cap. The suite itself was under 60 seconds and over the 20-second target, which is what this issue now tracks (retitled). https://git.eeqj.de/sneak/neoirc/pulls/267, which removes the waiting, is the fix; nothing is done about compile time. Model: opus-5-5
Sign in to join this conversation.
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/neoirc#263