From 0aaa94471f47cee4f67a59c4d1ede04d3bbdb6ba Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Wed, 7 Oct 2026 05:59:15 +0200 Subject: [PATCH] docs: README end-to-end limits match what the browser suites do (closes #293) The window.close override the issue named was removed with https://git.eeqj.de/sneak/AutistMask/issues/275, so it needs no entry. Every other place either suite changes or works around the shipped extension is now listed: the popup loaded in a tab, Chrome's wait before each site-connection request, its recording wrapper and click listener, its clipboard grant, two layout tests that write into the page, the forced leave during a decrypt, and Firefox's setting that forces the site-connection prompt into a window. Firefox's tolerated error is described as the leftover it is, with https://git.eeqj.de/sneak/AutistMask/issues/487 to remove it; the phishing blocklist is no longer listed as a failing fetch, and two stale figures are corrected. Model: opus-5-5 --- README.md | 113 ++++++++++++++++++++++++++++++++++-------------------- TODO.md | 14 +++++++ 2 files changed, 86 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index 79505b3..0254a3c 100644 --- a/README.md +++ b/README.md @@ -334,7 +334,9 @@ There are two suites, one per browser, and they share no code. Chrome runs on Playwright; Firefox has its own WebDriver client, because Playwright cannot observe errors on a Firefox extension page at all — see [Firefox](#firefox-make-test-e2e-firefox) below. Both require docker, and both -are outside `make check`. +are outside `make check`. Neither opens the popup from the toolbar button: both +load its page in an ordinary tab, so what the toolbar popup itself adds, its +size and its closing when it loses focus, is covered by neither. ### Chrome (`make test-e2e`) @@ -420,14 +422,36 @@ required to be present, so that check cannot pass by observing nothing. That last one is the standing floor under [#157](https://git.eeqj.de/sneak/AutistMask/issues/157). -Two limits of that coverage, neither of them papered over. The RPC is stubbed -throughout, so this is **not** a real dApp against a real network with real -funds; that remains a human pass before 1.0.0. The site-connection prompt is -raised through `chrome.action.openPopup()`, and headless Chromium's -browser-action popup is not a page Playwright can see or click, so that one -prompt is driven at the URL the extension itself puts on the action — the same -page and the same approval id, but whether a real toolbar click shows it is not -observable here. +The limits of that coverage and of the rest of the Chrome suite, none of them +papered over: + +- The RPC is stubbed throughout, so this is **not** a real dApp against a real + network with real funds; that remains a human pass before 1.0.0. +- The site-connection prompt is raised through `chrome.action.openPopup()`, and + headless Chromium's browser-action popup is not a page Playwright can see or + click, so that one prompt is driven at the URL the extension itself puts on + the action — the same page and the same approval id, but whether a real + toolbar click shows it is not observable here. +- Each site-connection request is made 1.5 seconds after the tab it will be + driven in is opened (`APPROVAL_TAB_SETTLE_MS` in `tests/e2e/run.js`), because + opening that tab closes the previous prompt's toolbar popup; a request made + just after that popup closes is not covered. +- In the tests that approve a signature or a transaction, the approval window + runs with `chrome.runtime.sendMessage` wrapped to record what it sends, and in + the tests that reject a site connection the prompt gets a click listener that + records that Reject was pressed; neither changes what the window does. +- The tap to copy test first grants clipboard permission to every page in the + browser, which the manifest does not ask for, so whether a real popup may + write to the clipboard on a click alone is not covered. +- The layout tests for an over-long flash message and for the password error + lines write the text straight into the page instead of letting the popup's + code put it there, and the second brings each screen up by toggling its + `hidden` class rather than navigating to it; they cover the layout, not the + code that fills it. +- Leaving the recovery phrase or private key screen while its decrypt runs is + forced by clicking Reveal and the settings gear in one page task, which a + person cannot do; the case a person can hit, the first decrypt after the popup + opens while libsodium is still loading, is not driven. Any test that drives a failure path on purpose declares the `console.error` it is about to provoke, via `errors.expect()`. That is not a mute: the declaration @@ -441,8 +465,8 @@ collecting for a fixed grace period after the last test returns the context. A request whose _first_ dispatch falls after that window is never seen at all and cannot fail the run. In practice a request a test fires without awaiting reaches the route handler about 10ms later, and anything on a repeating -timer gets observed on an earlier tick during the ~20s suite — but a one-shot -call deliberately deferred past the window will escape. +timer gets observed on an earlier tick during the suite — but a one-shot call +deliberately deferred past the window will escape. That interception covers the MV3 background service worker as well as the popup page, which it does not by default — `script/test-e2e` sets @@ -574,25 +598,26 @@ without an `await`. Demonstrated, not assumed: a `throw` placed past the first and on Chrome (`pageerror`), with the rest of the run unaffected because the approval view had already rendered. -One error is tolerated rather than fatal, listed in `ALLOWED_ERRORS` in -`tests/e2e/firefox/run.js` with the issue that will delete it, and printed on -every occurrence so the concession stays visible in the run output. It is -Firefox reporting the site-approval popup's unawaited `sendMessage` settling -after `window.close()` unloaded the context — the same teardown ordering as -[#275](https://git.eeqj.de/sneak/AutistMask/issues/275), and unsuppressable from -the calling code, because `BaseContext.wrapPromise` reports it whether or not a -handler is attached. Errors are read from the privileged `nsIConsoleService` in -Marionette's chrome context and filtered to non-warning entries whose -`sourceName` is the extension origin. That mechanism is not a stylistic choice. -WebDriver BiDi's `log.entryAdded` delivers **nothing** for extension pages: on a -plain `http://` page it reports uncaught errors with stack traces, and on the -`moz-extension://` popup it reports zero events, because Firefox's remote agent -excludes extension browsing contexts from BiDi observation. Any harness built on -Playwright-BiDi or Puppeteer-BiDi would therefore see nothing and report -success, which is exactly the vacuous check this repo has already shipped twice. -Do not migrate this suite to BiDi. +One error is still tolerated rather than fatal, listed in `ALLOWED_ERRORS` in +`tests/e2e/firefox/run.js` and printed on every occurrence so the concession +stays visible in the run output: Firefox reporting an extension promise that +settled after its page unloaded, from anywhere in the popup. Its cause was the +site-approval popup's unawaited `sendMessage` before `window.close()`, which +[#275](https://git.eeqj.de/sneak/AutistMask/issues/275) removed; Firefox runs +since have not printed it, and +[#487](https://git.eeqj.de/sneak/AutistMask/issues/487) removes the entry. +Errors are read from the privileged `nsIConsoleService` in Marionette's chrome +context and filtered to non-warning entries whose `sourceName` is the extension +origin. That mechanism is not a stylistic choice. WebDriver BiDi's +`log.entryAdded` delivers **nothing** for extension pages: on a plain `http://` +page it reports uncaught errors with stack traces, and on the `moz-extension://` +popup it reports zero events, because Firefox's remote agent excludes extension +browsing contexts from BiDi observation. Any harness built on Playwright-BiDi or +Puppeteer-BiDi would therefore see nothing and report success, which is exactly +the vacuous check this repo has already shipped twice. Do not migrate this suite +to BiDi. -Two limits are worth knowing, both real differences from the Chrome suite: +Three limits are worth knowing, all real differences from the Chrome suite: - **Error capture is poll-based, not event-streamed.** The console is drained at each step boundary, so an error is attributed to the step it was drained @@ -609,24 +634,30 @@ Two limits are worth knowing, both real differences from the Chrome suite: and silently evicts the oldest, so more than 250 console messages between two drains destroys the excess unread. 400 throws inside one step are reported as exactly the newest 250, three runs running. That buffer is shared with - Firefox's own console noise; a clean run peaks at 4 of 250 at the install - drain and 0 at every later drain, so the three steps here have wide headroom, - but a step that logs heavily could evict unread errors. What poll-based costs - is location, not coverage: an error cannot be placed within a step the way the - Chrome suite's `pageerror` events place it. + Firefox's own console noise; a clean run, measured when the suite had three + steps (popup load, wallet creation and Add Token), peaked at 4 of 250 at the + install drain and 0 at every later drain, but a step that logs heavily could + evict unread errors. What poll-based costs is location, not coverage: an error + cannot be placed within a step the way the Chrome suite's `pageerror` events + place it. - **Almost nothing is stubbed, which inverts the coverage of network-dependent code.** The container still runs with `--network none`, so the run is offline and no request can escape. The one thing it can reach is the loopback fixture in `tests/e2e/firefox/dapp.js`, which serves the dApp page and a JSON-RPC node and which the extension's `rpcUrl` is pointed at for the dApp steps; a JSON-RPC method that fixture does not model fails the run rather than - answering `null`. Everything else — Blockscout, the price feed, the phishing - blocklist — has no fixture and simply fails, and the extension swallows its - own fetch failures, so only the _failure_ branches of that code are ever - executed. A `ReferenceError` in the success path of `renderTransactions`, or - of price rendering, passes this suite green. The offline run is also weaker - than the Chrome suite's interception for those calls: it proves nothing got - out, but it cannot report which requests were attempted. + answering `null`. Everything else, Blockscout and the price feed among it, has + no fixture and simply fails, and the extension swallows its own fetch + failures, so only the _failure_ branches of that code are ever executed. A + `ReferenceError` in the success path of `renderTransactions`, or of price + rendering, passes this suite green. The offline run is also weaker than the + Chrome suite's interception for those calls: it proves nothing got out, but it + cannot report which requests were attempted. +- **The site-connection prompt always opens in a window of its own.** The + profile turns off `extensions.openPopupWithoutUserGesture.enabled`, so + `src/background/index.js` falls back from the toolbar popup, which WebDriver + cannot see, to `windows.create()`; the toolbar popup path is not covered on + Firefox. Neither `make test-e2e` nor `make test-e2e-firefox` is part of `make check` or `make test`. `REPO_POLICIES.md` caps `make test` at 60 seconds and a browser diff --git a/TODO.md b/TODO.md index cb85906..3871d10 100644 --- a/TODO.md +++ b/TODO.md @@ -45,6 +45,20 @@ but the review is broader than any of them. # Completed Steps +- 2026-10-07: The README's end-to-end limits now match what the two browser + suites do to the extension + ([#293](https://git.eeqj.de/sneak/AutistMask/issues/293)). The `window.close` + override the issue named went with + [#275](https://git.eeqj.de/sneak/AutistMask/issues/275), so it needs no entry. + Added: both suites load the popup in a tab rather than from the toolbar; + Chrome waits 1.5 seconds before each site-connection request, records what + approval windows send and click, grants clipboard permission, writes text + straight into the page in two layout tests, and forces the leave during a + decrypt; Firefox forces the site-connection prompt into a window. Corrected: + Firefox's one tolerated error, whose cause that fix removed (the entry goes in + [#487](https://git.eeqj.de/sneak/AutistMask/issues/487)), the phishing + blocklist the extension no longer fetches, and two stale figures. + - 2026-10-07: Back from Settings no longer shows Settings again after the settings gear was pressed on the recovery phrase or a delete wallet screen opened from Settings, with or without a reopen in between