test: make e2e error attribution total, and fix the README canary text
All checks were successful
check / check (push) Successful in 19s

The error collector had a window API (mark/since) and twice a record fell
outside somebody's window and was silently dropped, producing a green run
that proved nothing: first the mark started after test 1, discarding
everything recorded during launch; then the tail after the final test was
never read at all, so a request escaping the fixtures at the end of the
last test reported 5/5 passed and exit 0.

Rather than patch a second boundary and invite a third, the window
concept is gone. ErrorCollector exposes only take(), which always drains
everything outstanding, so successive takes partition the whole record
stream with no gaps, and seal(), which closes the stream at the end of
the run and routes stragglers straight to a failure. Attribution is
total by construction: launch through test 1 goes to test 1, each
subsequent interval to the test that ends it, the tail to the suite.

The tail also needs to exist before it can be drained. A request a test
fires without awaiting reaches the route handler about 10ms after that
test's function resolves, and closing the context does not wait for it,
so with no window at all it died unobserved. The run now keeps
collecting for a bounded 1.5s after the last test before teardown.

Also:

- README described a canary that was built, found to kill the service
  worker, and deleted. Replaced with what actually runs: the harness
  waits for the background worker's own startup blocklist fetch to reach
  the route handler and aborts if it does not. Documents
  --host-resolver-rules=MAP * ~NOTFOUND as defence in depth.
- The canary's failure message asserted traffic was escaping to the real
  internet and blamed the -e flag. It cannot distinguish that from a lost
  startup race, so it now states what was observed and lists both causes.
- E2E_TRACE_NETWORK was compared strictly to "1", so E2E_TRACE_NETWORK=true
  silently did nothing. Recognised on/off values are accepted and anything
  else is a hard error rather than a quiet default.
- The measured margin that makes the canary sound is route install at
  11-23ms against the worker fetch at 525-883ms, not the 30s timeout
  slack the comment cited.
This commit is contained in:
clawbot
2026-08-09 15:50:45 +00:00
parent a3075f2f47
commit a13862d991
4 changed files with 187 additions and 36 deletions

View File

@@ -39,23 +39,55 @@ function isAllowed(text) {
return ALLOWED_ERRORS.some((a) => a.pattern.test(text));
}
// Collects every uncaught page error, console.error and unstubbed
// request, and hands each one to exactly one reporter.
//
// This deliberately has NO window API. It used to expose mark()/since()
// so a test could ask for "the errors since I started", and that shape
// produced a green run that proved nothing twice over: first the mark
// started after test 1, so everything recorded during launch was
// discarded, then the tail after the final test was never read at all. In
// both cases a record fell outside somebody's window and vanished, which
// is the precise failure this harness exists to prevent.
//
// So there is no window left to fall outside of. take() is the only
// reader and it always takes everything outstanding, so successive takes
// partition the entire record stream with no gaps. seal() closes the
// stream once the run is over and routes anything later straight to a
// callback rather than into a list nobody reads again. Attribution is
// therefore total by construction, and the runner turns every attributed
// record into a failure.
class ErrorCollector {
constructor() {
this.entries = [];
this.taken = 0;
this.onLate = null;
}
record(kind, text) {
const line = kind + ": " + String(text).split("\n")[0];
if (isAllowed(line)) return;
if (this.onLate) {
// Sealed: no test and no suite phase is left to attribute
// this to, so hand it over now instead of accumulating it
// where nothing will look.
this.onLate(line);
return;
}
this.entries.push(line);
}
mark() {
return this.entries.length;
// Everything recorded since the previous take(). Never yields a
// record twice and never skips one.
take() {
const out = this.entries.slice(this.taken);
this.taken = this.entries.length;
return out;
}
since(mark) {
return this.entries.slice(mark);
// Close the stream: later records go to onLate instead of the list.
seal(onLate) {
this.onLate = onLate;
}
}
@@ -89,8 +121,16 @@ async function serviceWorker(ctx) {
}
// How long to wait for the background worker's first outbound request.
// Measured at roughly 650ms after the route is installed; the margin is
// for a loaded machine, not for hope.
//
// The margin that actually decides whether this check is sound is not
// this timeout — it is whether the route handler is installed before the
// worker fetches. Measured over several runs: route installation
// completes 11-23ms after the context comes up, and the worker's
// blocklist fetch arrives 525-883ms after that, so the route wins by
// roughly 25-50x. This 30s figure is only slack for a loaded machine on
// top of that; losing the race fails the run rather than passing it
// quietly, which was verified by forcing a 3s delay before route
// installation.
const WORKER_TRAFFIC_TIMEOUT_MS = 30000;
// ctx.route() only sees service-worker requests when Playwright runs with
@@ -115,19 +155,29 @@ async function assertWorkerTrafficIntercepted(stubs) {
);
if (seen) return seen;
// State the observation, not a conclusion. This fires for at least
// two quite different causes and the harness cannot tell them apart
// from here, so guessing one of them in the message sends the reader
// the wrong way.
throw new Error(
"no service-worker request reached the route handler within " +
"observed no service-worker request in the route handler within " +
WORKER_TRAFFIC_TIMEOUT_MS +
"ms, so background worker traffic is escaping this harness and " +
"going to the real internet. Run the suite through " +
"script/test-e2e, which sets " +
"PW_EXPERIMENTAL_SERVICE_WORKER_NETWORK_EVENTS=1. If a " +
"Playwright upgrade dropped that flag, replace the mechanism or " +
"downgrade the isolation claims in tests/e2e/network.js and " +
"README.md — do not delete this check. If instead the " +
"background worker legitimately stopped making startup " +
"requests, this check needs a new anchor, because there is no " +
"longer any worker traffic to observe",
"ms. Under working interception the background worker's " +
"startup blocklist fetch (src/background/index.js) reaches the " +
"handler about half a second after the route is installed. " +
"Two causes are plausible and this check cannot distinguish " +
"them: (1) service-worker interception is not in effect, so " +
"that traffic went to the real internet unobserved — the suite " +
"must be run through script/test-e2e, which sets " +
"PW_EXPERIMENTAL_SERVICE_WORKER_NETWORK_EVENTS=1, and a " +
"Playwright upgrade may have dropped or renamed that flag; " +
"(2) no worker request was made in the first place — the route " +
"lost the startup race, or the worker no longer fetches at " +
"startup, in which case this check needs a new anchor because " +
"there is no longer any worker traffic to observe. Either way " +
"the fix is a replacement mechanism or an honest downgrade of " +
"the isolation claims in tests/e2e/network.js and README.md — " +
"not deleting this check",
);
}