docs: README end-to-end limits match what the browser suites do (closes #293)
The window.close override the issue named was removed with #275, so it needs no entry. Every other place either suite changes or works around the shipped extension is now listed: the popup loaded in a tab, Chrome's wait before each site-connection request, its recording wrapper and click listener, its clipboard grant, two layout tests that write into the page, the forced leave during a decrypt, and Firefox's setting that forces the site-connection prompt into a window. Firefox's tolerated error is described as the leftover it is, with #487 to remove it; the phishing blocklist is no longer listed as a failing fetch, and two stale figures are corrected. Model: opus-5-5
This commit is contained in:
@@ -334,7 +334,9 @@ There are two suites, one per browser, and they share no code. Chrome runs on
|
||||
Playwright; Firefox has its own WebDriver client, because Playwright cannot
|
||||
observe errors on a Firefox extension page at all — see
|
||||
[Firefox](#firefox-make-test-e2e-firefox) below. Both require docker, and both
|
||||
are outside `make check`.
|
||||
are outside `make check`. Neither opens the popup from the toolbar button: both
|
||||
load its page in an ordinary tab, so what the toolbar popup itself adds, its
|
||||
size and its closing when it loses focus, is covered by neither.
|
||||
|
||||
### Chrome (`make test-e2e`)
|
||||
|
||||
@@ -420,14 +422,36 @@ required to be present, so that check cannot pass by observing nothing. That
|
||||
last one is the standing floor under
|
||||
[#157](https://git.eeqj.de/sneak/AutistMask/issues/157).
|
||||
|
||||
Two limits of that coverage, neither of them papered over. The RPC is stubbed
|
||||
throughout, so this is **not** a real dApp against a real network with real
|
||||
funds; that remains a human pass before 1.0.0. The site-connection prompt is
|
||||
raised through `chrome.action.openPopup()`, and headless Chromium's
|
||||
browser-action popup is not a page Playwright can see or click, so that one
|
||||
prompt is driven at the URL the extension itself puts on the action — the same
|
||||
page and the same approval id, but whether a real toolbar click shows it is not
|
||||
observable here.
|
||||
The limits of that coverage and of the rest of the Chrome suite, none of them
|
||||
papered over:
|
||||
|
||||
- The RPC is stubbed throughout, so this is **not** a real dApp against a real
|
||||
network with real funds; that remains a human pass before 1.0.0.
|
||||
- The site-connection prompt is raised through `chrome.action.openPopup()`, and
|
||||
headless Chromium's browser-action popup is not a page Playwright can see or
|
||||
click, so that one prompt is driven at the URL the extension itself puts on
|
||||
the action — the same page and the same approval id, but whether a real
|
||||
toolbar click shows it is not observable here.
|
||||
- Each site-connection request is made 1.5 seconds after the tab it will be
|
||||
driven in is opened (`APPROVAL_TAB_SETTLE_MS` in `tests/e2e/run.js`), because
|
||||
opening that tab closes the previous prompt's toolbar popup; a request made
|
||||
just after that popup closes is not covered.
|
||||
- In the tests that approve a signature or a transaction, the approval window
|
||||
runs with `chrome.runtime.sendMessage` wrapped to record what it sends, and in
|
||||
the tests that reject a site connection the prompt gets a click listener that
|
||||
records that Reject was pressed; neither changes what the window does.
|
||||
- The tap to copy test first grants clipboard permission to every page in the
|
||||
browser, which the manifest does not ask for, so whether a real popup may
|
||||
write to the clipboard on a click alone is not covered.
|
||||
- The layout tests for an over-long flash message and for the password error
|
||||
lines write the text straight into the page instead of letting the popup's
|
||||
code put it there, and the second brings each screen up by toggling its
|
||||
`hidden` class rather than navigating to it; they cover the layout, not the
|
||||
code that fills it.
|
||||
- Leaving the recovery phrase or private key screen while its decrypt runs is
|
||||
forced by clicking Reveal and the settings gear in one page task, which a
|
||||
person cannot do; the case a person can hit, the first decrypt after the popup
|
||||
opens while libsodium is still loading, is not driven.
|
||||
|
||||
Any test that drives a failure path on purpose declares the `console.error` it
|
||||
is about to provoke, via `errors.expect()`. That is not a mute: the declaration
|
||||
@@ -441,8 +465,8 @@ collecting for a fixed grace period after the last test returns
|
||||
the context. A request whose _first_ dispatch falls after that window is never
|
||||
seen at all and cannot fail the run. In practice a request a test fires without
|
||||
awaiting reaches the route handler about 10ms later, and anything on a repeating
|
||||
timer gets observed on an earlier tick during the ~20s suite — but a one-shot
|
||||
call deliberately deferred past the window will escape.
|
||||
timer gets observed on an earlier tick during the suite — but a one-shot call
|
||||
deliberately deferred past the window will escape.
|
||||
|
||||
That interception covers the MV3 background service worker as well as the popup
|
||||
page, which it does not by default — `script/test-e2e` sets
|
||||
@@ -574,25 +598,26 @@ without an `await`. Demonstrated, not assumed: a `throw` placed past the first
|
||||
and on Chrome (`pageerror`), with the rest of the run unaffected because the
|
||||
approval view had already rendered.
|
||||
|
||||
One error is tolerated rather than fatal, listed in `ALLOWED_ERRORS` in
|
||||
`tests/e2e/firefox/run.js` with the issue that will delete it, and printed on
|
||||
every occurrence so the concession stays visible in the run output. It is
|
||||
Firefox reporting the site-approval popup's unawaited `sendMessage` settling
|
||||
after `window.close()` unloaded the context — the same teardown ordering as
|
||||
[#275](https://git.eeqj.de/sneak/AutistMask/issues/275), and unsuppressable from
|
||||
the calling code, because `BaseContext.wrapPromise` reports it whether or not a
|
||||
handler is attached. Errors are read from the privileged `nsIConsoleService` in
|
||||
Marionette's chrome context and filtered to non-warning entries whose
|
||||
`sourceName` is the extension origin. That mechanism is not a stylistic choice.
|
||||
WebDriver BiDi's `log.entryAdded` delivers **nothing** for extension pages: on a
|
||||
plain `http://` page it reports uncaught errors with stack traces, and on the
|
||||
`moz-extension://` popup it reports zero events, because Firefox's remote agent
|
||||
excludes extension browsing contexts from BiDi observation. Any harness built on
|
||||
Playwright-BiDi or Puppeteer-BiDi would therefore see nothing and report
|
||||
success, which is exactly the vacuous check this repo has already shipped twice.
|
||||
Do not migrate this suite to BiDi.
|
||||
One error is still tolerated rather than fatal, listed in `ALLOWED_ERRORS` in
|
||||
`tests/e2e/firefox/run.js` and printed on every occurrence so the concession
|
||||
stays visible in the run output: Firefox reporting an extension promise that
|
||||
settled after its page unloaded, from anywhere in the popup. Its cause was the
|
||||
site-approval popup's unawaited `sendMessage` before `window.close()`, which
|
||||
[#275](https://git.eeqj.de/sneak/AutistMask/issues/275) removed; Firefox runs
|
||||
since have not printed it, and
|
||||
[#487](https://git.eeqj.de/sneak/AutistMask/issues/487) removes the entry.
|
||||
Errors are read from the privileged `nsIConsoleService` in Marionette's chrome
|
||||
context and filtered to non-warning entries whose `sourceName` is the extension
|
||||
origin. That mechanism is not a stylistic choice. WebDriver BiDi's
|
||||
`log.entryAdded` delivers **nothing** for extension pages: on a plain `http://`
|
||||
page it reports uncaught errors with stack traces, and on the `moz-extension://`
|
||||
popup it reports zero events, because Firefox's remote agent excludes extension
|
||||
browsing contexts from BiDi observation. Any harness built on Playwright-BiDi or
|
||||
Puppeteer-BiDi would therefore see nothing and report success, which is exactly
|
||||
the vacuous check this repo has already shipped twice. Do not migrate this suite
|
||||
to BiDi.
|
||||
|
||||
Two limits are worth knowing, both real differences from the Chrome suite:
|
||||
Three limits are worth knowing, all real differences from the Chrome suite:
|
||||
|
||||
- **Error capture is poll-based, not event-streamed.** The console is drained at
|
||||
each step boundary, so an error is attributed to the step it was drained
|
||||
@@ -609,24 +634,29 @@ Two limits are worth knowing, both real differences from the Chrome suite:
|
||||
and silently evicts the oldest, so more than 250 console messages between two
|
||||
drains destroys the excess unread. 400 throws inside one step are reported as
|
||||
exactly the newest 250, three runs running. That buffer is shared with
|
||||
Firefox's own console noise; a clean run peaks at 4 of 250 at the install
|
||||
drain and 0 at every later drain, so the three steps here have wide headroom,
|
||||
but a step that logs heavily could evict unread errors. What poll-based costs
|
||||
is location, not coverage: an error cannot be placed within a step the way the
|
||||
Chrome suite's `pageerror` events place it.
|
||||
Firefox's own console noise; a clean run of the suite's first three steps,
|
||||
measured before the rest were added, peaked at 4 of 250 at the install drain
|
||||
and 0 at every later drain, but a step that logs heavily could evict unread
|
||||
errors. What poll-based costs is location, not coverage: an error cannot be
|
||||
placed within a step the way the Chrome suite's `pageerror` events place it.
|
||||
- **Almost nothing is stubbed, which inverts the coverage of network-dependent
|
||||
code.** The container still runs with `--network none`, so the run is offline
|
||||
and no request can escape. The one thing it can reach is the loopback fixture
|
||||
in `tests/e2e/firefox/dapp.js`, which serves the dApp page and a JSON-RPC node
|
||||
and which the extension's `rpcUrl` is pointed at for the dApp steps; a
|
||||
JSON-RPC method that fixture does not model fails the run rather than
|
||||
answering `null`. Everything else — Blockscout, the price feed, the phishing
|
||||
blocklist — has no fixture and simply fails, and the extension swallows its
|
||||
own fetch failures, so only the _failure_ branches of that code are ever
|
||||
executed. A `ReferenceError` in the success path of `renderTransactions`, or
|
||||
of price rendering, passes this suite green. The offline run is also weaker
|
||||
than the Chrome suite's interception for those calls: it proves nothing got
|
||||
out, but it cannot report which requests were attempted.
|
||||
answering `null`. Everything else, Blockscout and the price feed among it, has
|
||||
no fixture and simply fails, and the extension swallows its own fetch
|
||||
failures, so only the _failure_ branches of that code are ever executed. A
|
||||
`ReferenceError` in the success path of `renderTransactions`, or of price
|
||||
rendering, passes this suite green. The offline run is also weaker than the
|
||||
Chrome suite's interception for those calls: it proves nothing got out, but it
|
||||
cannot report which requests were attempted.
|
||||
- **The site-connection prompt always opens in a window of its own.** The
|
||||
profile turns off `extensions.openPopupWithoutUserGesture.enabled`, so
|
||||
`src/background/index.js` falls back from the toolbar popup, which WebDriver
|
||||
cannot see, to `windows.create()`; the toolbar popup path is not covered on
|
||||
Firefox.
|
||||
|
||||
Neither `make test-e2e` nor `make test-e2e-firefox` is part of `make check` or
|
||||
`make test`. `REPO_POLICIES.md` caps `make test` at 60 seconds and a browser
|
||||
|
||||
Reference in New Issue
Block a user