test: make e2e error attribution total, and fix the README canary text
All checks were successful
check / check (push) Successful in 19s
All checks were successful
check / check (push) Successful in 19s
The error collector had a window API (mark/since) and twice a record fell outside somebody's window and was silently dropped, producing a green run that proved nothing: first the mark started after test 1, discarding everything recorded during launch; then the tail after the final test was never read at all, so a request escaping the fixtures at the end of the last test reported 5/5 passed and exit 0. Rather than patch a second boundary and invite a third, the window concept is gone. ErrorCollector exposes only take(), which always drains everything outstanding, so successive takes partition the whole record stream with no gaps, and seal(), which closes the stream at the end of the run and routes stragglers straight to a failure. Attribution is total by construction: launch through test 1 goes to test 1, each subsequent interval to the test that ends it, the tail to the suite. The tail also needs to exist before it can be drained. A request a test fires without awaiting reaches the route handler about 10ms after that test's function resolves, and closing the context does not wait for it, so with no window at all it died unobserved. The run now keeps collecting for a bounded 1.5s after the last test before teardown. Also: - README described a canary that was built, found to kill the service worker, and deleted. Replaced with what actually runs: the harness waits for the background worker's own startup blocklist fetch to reach the route handler and aborts if it does not. Documents --host-resolver-rules=MAP * ~NOTFOUND as defence in depth. - The canary's failure message asserted traffic was escaping to the real internet and blamed the -e flag. It cannot distinguish that from a lost startup race, so it now states what was observed and lists both causes. - E2E_TRACE_NETWORK was compared strictly to "1", so E2E_TRACE_NETWORK=true silently did nothing. Recognised on/off values are accepted and anything else is a hard error rather than a quiet default. - The measured margin that makes the canary sound is route install at 11-23ms against the worker fetch at 525-883ms, not the 30s timeout slack the comment cited.
This commit is contained in:
@@ -39,23 +39,55 @@ function isAllowed(text) {
|
||||
return ALLOWED_ERRORS.some((a) => a.pattern.test(text));
|
||||
}
|
||||
|
||||
// Collects every uncaught page error, console.error and unstubbed
|
||||
// request, and hands each one to exactly one reporter.
|
||||
//
|
||||
// This deliberately has NO window API. It used to expose mark()/since()
|
||||
// so a test could ask for "the errors since I started", and that shape
|
||||
// produced a green run that proved nothing twice over: first the mark
|
||||
// started after test 1, so everything recorded during launch was
|
||||
// discarded, then the tail after the final test was never read at all. In
|
||||
// both cases a record fell outside somebody's window and vanished, which
|
||||
// is the precise failure this harness exists to prevent.
|
||||
//
|
||||
// So there is no window left to fall outside of. take() is the only
|
||||
// reader and it always takes everything outstanding, so successive takes
|
||||
// partition the entire record stream with no gaps. seal() closes the
|
||||
// stream once the run is over and routes anything later straight to a
|
||||
// callback rather than into a list nobody reads again. Attribution is
|
||||
// therefore total by construction, and the runner turns every attributed
|
||||
// record into a failure.
|
||||
class ErrorCollector {
|
||||
constructor() {
|
||||
this.entries = [];
|
||||
this.taken = 0;
|
||||
this.onLate = null;
|
||||
}
|
||||
|
||||
record(kind, text) {
|
||||
const line = kind + ": " + String(text).split("\n")[0];
|
||||
if (isAllowed(line)) return;
|
||||
if (this.onLate) {
|
||||
// Sealed: no test and no suite phase is left to attribute
|
||||
// this to, so hand it over now instead of accumulating it
|
||||
// where nothing will look.
|
||||
this.onLate(line);
|
||||
return;
|
||||
}
|
||||
this.entries.push(line);
|
||||
}
|
||||
|
||||
mark() {
|
||||
return this.entries.length;
|
||||
// Everything recorded since the previous take(). Never yields a
|
||||
// record twice and never skips one.
|
||||
take() {
|
||||
const out = this.entries.slice(this.taken);
|
||||
this.taken = this.entries.length;
|
||||
return out;
|
||||
}
|
||||
|
||||
since(mark) {
|
||||
return this.entries.slice(mark);
|
||||
// Close the stream: later records go to onLate instead of the list.
|
||||
seal(onLate) {
|
||||
this.onLate = onLate;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -89,8 +121,16 @@ async function serviceWorker(ctx) {
|
||||
}
|
||||
|
||||
// How long to wait for the background worker's first outbound request.
|
||||
// Measured at roughly 650ms after the route is installed; the margin is
|
||||
// for a loaded machine, not for hope.
|
||||
//
|
||||
// The margin that actually decides whether this check is sound is not
|
||||
// this timeout — it is whether the route handler is installed before the
|
||||
// worker fetches. Measured over several runs: route installation
|
||||
// completes 11-23ms after the context comes up, and the worker's
|
||||
// blocklist fetch arrives 525-883ms after that, so the route wins by
|
||||
// roughly 25-50x. This 30s figure is only slack for a loaded machine on
|
||||
// top of that; losing the race fails the run rather than passing it
|
||||
// quietly, which was verified by forcing a 3s delay before route
|
||||
// installation.
|
||||
const WORKER_TRAFFIC_TIMEOUT_MS = 30000;
|
||||
|
||||
// ctx.route() only sees service-worker requests when Playwright runs with
|
||||
@@ -115,19 +155,29 @@ async function assertWorkerTrafficIntercepted(stubs) {
|
||||
);
|
||||
if (seen) return seen;
|
||||
|
||||
// State the observation, not a conclusion. This fires for at least
|
||||
// two quite different causes and the harness cannot tell them apart
|
||||
// from here, so guessing one of them in the message sends the reader
|
||||
// the wrong way.
|
||||
throw new Error(
|
||||
"no service-worker request reached the route handler within " +
|
||||
"observed no service-worker request in the route handler within " +
|
||||
WORKER_TRAFFIC_TIMEOUT_MS +
|
||||
"ms, so background worker traffic is escaping this harness and " +
|
||||
"going to the real internet. Run the suite through " +
|
||||
"script/test-e2e, which sets " +
|
||||
"PW_EXPERIMENTAL_SERVICE_WORKER_NETWORK_EVENTS=1. If a " +
|
||||
"Playwright upgrade dropped that flag, replace the mechanism or " +
|
||||
"downgrade the isolation claims in tests/e2e/network.js and " +
|
||||
"README.md — do not delete this check. If instead the " +
|
||||
"background worker legitimately stopped making startup " +
|
||||
"requests, this check needs a new anchor, because there is no " +
|
||||
"longer any worker traffic to observe",
|
||||
"ms. Under working interception the background worker's " +
|
||||
"startup blocklist fetch (src/background/index.js) reaches the " +
|
||||
"handler about half a second after the route is installed. " +
|
||||
"Two causes are plausible and this check cannot distinguish " +
|
||||
"them: (1) service-worker interception is not in effect, so " +
|
||||
"that traffic went to the real internet unobserved — the suite " +
|
||||
"must be run through script/test-e2e, which sets " +
|
||||
"PW_EXPERIMENTAL_SERVICE_WORKER_NETWORK_EVENTS=1, and a " +
|
||||
"Playwright upgrade may have dropped or renamed that flag; " +
|
||||
"(2) no worker request was made in the first place — the route " +
|
||||
"lost the startup race, or the worker no longer fetches at " +
|
||||
"startup, in which case this check needs a new anchor because " +
|
||||
"there is no longer any worker traffic to observe. Either way " +
|
||||
"the fix is a replacement mechanism or an honest downgrade of " +
|
||||
"the isolation claims in tests/e2e/network.js and README.md — " +
|
||||
"not deleting this check",
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user