sneak/AutistMask
1
0
Fork 0
Code Issues 42 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

decision: no browser in the agent environment, so several 1.0.0 items cannot have their DoD verified #173

New Issue
Open
opened 2026-08-09 04:54:11 +02:00 by clawbot · 2 comments
clawbot commented 2026-08-09 04:54:11 +02:00
Collaborator
Copy Link

Structural gap on the path to 1.0.0 that I can't resolve on my own, surfacing
it early rather than at tag time.

The problem

This is a browser extension, and a meaningful share of the 1.0.0 milestone can
only be genuinely verified by loading the built artifact into a browser and
driving it. There is no browser in the agent environment, so no agent can
perform that step. Automated coverage is unit tests only.

This already bit the two units driven so far:

  • #157 / PR #171 (plaintext password across the messaging boundary) —
    DoD items 2-5 require live dApp eth_sendTransaction, personal_sign and
    eth_signTypedData_v4 round trips. The implementer stated plainly that
    Chrome was not verified interactively and did not claim the build as proof.
    The change is to a wallet's signing path, covered by 13 unit tests and a
    round-trip test, but not by a single real transaction.
  • #149 / PR #169 — verified at the artifact level by grepping bundles,
    which was sufficient there because the property was a compile-time constant.
    That approach does not generalise.

Items still ahead with the same exposure: #153 (Firefox target
non-functional — its DoD is six manual browser checks across two browsers),
#158 (MV3 service-worker termination, which requires letting a real worker
idle out), #150 and #151 (the two unreachable-screen crashes, where the
whole point is that a screen opens), #155 (WaitTx timing), #162 and
#161 (new UI flows).

So a large fraction of the milestone will otherwise reach merge-ready with
"unit tests pass, browser behaviour unverified" as the honest state. For most
repos that is a quality question. For a wallet that signs transactions, I don't
think I should be the one deciding it is acceptable.

Options

  • (a) You do the manual QA passes. I keep driving units to merge-ready
    with automated coverage, and each PR carries an explicit, itemised list of
    what still needs a human browser pass. You verify before merge, or in a batch
    before the 1.0.0 tag. Zero new dependencies and zero new infrastructure; the
    cost is your time, concentrated on the riskiest changes.
  • (b) Add browser automation. Playwright can load an unpacked extension in
    a persistent Chromium context and drive the popup, which would cover most of
    the DoD items above and would keep covering them as regressions. Two real
    costs: it is a substantial dev dependency for a project whose whole ethos is
    minimal dependencies, and it cannot live in make test — REPO_POLICIES caps
    that at 20 seconds with a 30-second timeout, and browser tests will not fit.
    It would need its own target (make test-e2e) and its own place in CI, and
    Firefox MV2 coverage would be a second, separate effort.
  • (c) Accept unit-test-only coverage for 1.0. Tag with the gap documented.
    Cheapest, and I'd advise against it for a wallet.

My recommendation

(a) now, (b) as a post-1.0 issue. Manual passes from you are the honest
answer for the handful of genuinely interactive items still in the milestone,
and they cost nothing structurally. Standing up Playwright mid-milestone would
be a large detour that delays the security fixes actually queued, and the
minimal-dependency principle in the README is a real constraint rather than a
slogan. Once 1.0 is out and the surface stops moving, (b) becomes much better
value.

If you pick (a): tell me whether you want to verify per-PR before each merge,
or in one batch before the tag. Per-PR is safer and catches problems while the
author's context is fresh; batching is less interruption for you. I'd suggest
per-PR for the signing-path and Firefox changes (#157, #153) and batching the
rest.

If you pick (b): I'll file it as its own issue and sequence it before the items
that depend on it, and I'll need your call on whether it blocks the 1.0.0 tag
or lands alongside it.

Immediate decision needed on one PR

PR #171 (#157, the password/messaging fix) is in review now and will land
in this state — signing-path change, strong unit coverage, no interactive
verification. Whatever you choose above, that specific PR is the first one
where it bites. If you want a manual pass on anything, that is the one to pick.

Not blocking anything else; I am continuing down the milestone.

Structural gap on the path to 1.0.0 that I can't resolve on my own, surfacing it early rather than at tag time. ## The problem This is a browser extension, and a meaningful share of the 1.0.0 milestone can only be genuinely verified by loading the built artifact into a browser and driving it. There is no browser in the agent environment, so no agent can perform that step. Automated coverage is unit tests only. This already bit the two units driven so far: - **#157 / PR #171** (plaintext password across the messaging boundary) — DoD items 2-5 require live dApp `eth_sendTransaction`, `personal_sign` and `eth_signTypedData_v4` round trips. The implementer stated plainly that Chrome was not verified interactively and did not claim the build as proof. The change is to a wallet's **signing path**, covered by 13 unit tests and a round-trip test, but not by a single real transaction. - **#149 / PR #169** — verified at the artifact level by grepping bundles, which was sufficient there because the property was a compile-time constant. That approach does not generalise. Items still ahead with the same exposure: **#153** (Firefox target non-functional — its DoD is six manual browser checks across two browsers), **#158** (MV3 service-worker termination, which requires letting a real worker idle out), **#150** and **#151** (the two unreachable-screen crashes, where the whole point is that a screen opens), **#155** (WaitTx timing), **#162** and **#161** (new UI flows). So a large fraction of the milestone will otherwise reach `merge-ready` with "unit tests pass, browser behaviour unverified" as the honest state. For most repos that is a quality question. For a wallet that signs transactions, I don't think I should be the one deciding it is acceptable. ## Options - **(a) You do the manual QA passes.** I keep driving units to `merge-ready` with automated coverage, and each PR carries an explicit, itemised list of what still needs a human browser pass. You verify before merge, or in a batch before the 1.0.0 tag. Zero new dependencies and zero new infrastructure; the cost is your time, concentrated on the riskiest changes. - **(b) Add browser automation.** Playwright can load an unpacked extension in a persistent Chromium context and drive the popup, which would cover most of the DoD items above and would keep covering them as regressions. Two real costs: it is a substantial dev dependency for a project whose whole ethos is minimal dependencies, and it cannot live in `make test` — REPO_POLICIES caps that at 20 seconds with a 30-second timeout, and browser tests will not fit. It would need its own target (`make test-e2e`) and its own place in CI, and Firefox MV2 coverage would be a second, separate effort. - **(c) Accept unit-test-only coverage for 1.0.** Tag with the gap documented. Cheapest, and I'd advise against it for a wallet. ## My recommendation **(a) now, (b) as a post-1.0 issue.** Manual passes from you are the honest answer for the handful of genuinely interactive items still in the milestone, and they cost nothing structurally. Standing up Playwright mid-milestone would be a large detour that delays the security fixes actually queued, and the minimal-dependency principle in the README is a real constraint rather than a slogan. Once 1.0 is out and the surface stops moving, (b) becomes much better value. If you pick (a): tell me whether you want to verify per-PR before each merge, or in one batch before the tag. Per-PR is safer and catches problems while the author's context is fresh; batching is less interruption for you. I'd suggest per-PR for the signing-path and Firefox changes (#157, #153) and batching the rest. If you pick (b): I'll file it as its own issue and sequence it before the items that depend on it, and I'll need your call on whether it blocks the 1.0.0 tag or lands alongside it. ## Immediate decision needed on one PR **PR #171** (#157, the password/messaging fix) is in review now and will land in this state — signing-path change, strong unit coverage, no interactive verification. Whatever you choose above, that specific PR is the first one where it bites. If you want a manual pass on anything, that is the one to pick. Not blocking anything else; I am continuing down the milestone.
sneak was assigned by clawbot 2026-08-09 04:54:11 +02:00
clawbot commented 2026-08-09 16:09:35 +02:00
Author
Collaborator
Copy Link

Correction: the central premise of this issue is false

I wrote this issue from an inference about the environment rather than a
measurement of it. I have now measured it. There is a browser available, it
runs in a container, and it can drive this extension. Everything below is
observed output, not a plan.

What I actually ran

Image, pinned by digest:

mcr.microsoft.com/playwright@sha256:35246d87a7c88ea9b771c65d33171b2611b02a8253b4b12ce6f94376c55f99f2

with playwright-core@1.56.0, launching the unpacked Chrome MV3 build:

chromium.launchPersistentContext(userDir, {
    channel: "chromium",      // NOT the default headless shell - that one
    headless: true,           // silently refuses to load extensions
    args: [
        `--disable-extensions-except=${extPath}`,
        `--load-extension=${extPath}`,
        "--no-sandbox",
    ],
});

Result on the first run against dist/chrome/:

EXTENSION_ID=cieocojfinnamfiijllmlebfjdkedmfp
POPUP_TITLE=AutistMask
VISIBLE_VIEWS=["view-welcome"]
BODY_TEXT_SNIPPET="DEBUG / INSECURE (welcome)\nAutistMask by @sneak\n..."

The service worker starts, the extension id is discoverable from its URL, and
chrome-extension://<id>/src/popup/index.html renders and is fully
scriptable.

It survives a real end-to-end flow

Driving the actual UI - #btn-welcome-add, #btn-generate-phrase, fill
password, #btn-add-wallet-confirm - produces a real wallet and lands on
#view-main. That exercises BIP-39 generation, libsodium vault encryption and
extension storage persistence for real, in a real browser, not in jsdom.

The part that matters: I proved the harness can FAIL

A harness that has never been shown to fail is worth nothing, and this repo has
already shipped two vacuous checks that passed on things they could not detect.
So I pointed it at a bug that is live on main right now - #150, where
src/popup/views/addToken.js line 24 calls showView(...) without importing
it. I did not have to plant anything.

STEP wallet-created OK
STEP address-detail OK
CHECK_150 addTokenVisible=false errors=["pageerror: showView is not defined"]

The harness caught it: the screen did not open, and the exact ReferenceError
was captured. make check is green on this same tree. That is the entire
argument for the harness in one line - make check passes, the screen is
broken, and only the browser knows.

It also found a bug nobody had reported

Every popup load emits an uncaught error:

pageerror: Aborted(CompileError: WebAssembly.instantiate(): Refused to compile
or instantiate WebAssembly module because neither 'wasm-eval' nor 'unsafe-eval'
is an allowed source of script in the following Content Security Policy
directive: "script-src 'self'")

libsodium's WASM build is refused by the extension CSP and falls back to
asm.js. Wallet creation still works, so this is not fatal, but a wallet's
crypto library silently failing its preferred backend on every single load is
not something to discover at 1.0. Filed separately.

Also observed: the popup makes outbound API calls that returned 401 in the
container. The harness must intercept network at the browser level so it is
deterministic and offline, rather than depending on third-party endpoints.

Corrected coverage assessment

Reassessing the exposure list from the issue body against what was measured,
rather than against what I assumed:

Item Verifiable by the Chrome harness?
#150 AddToken unreachable Yes - already demonstrated above.
#151 TransactionDetail crash Yes - same mechanism; needs a stubbed Blockscout response to seed an ERC-20 tx, which is exactly why network interception belongs in the harness.
#155 WaitTx 60s timeout overwrites success Yes - clock and network are both controllable from the harness; this becomes a deterministic test rather than a 60-second wait.
#161 show recovery phrase Yes - password-gated popup flow, fully drivable.
#162 delete an address from an HD wallet Yes - same.
#158 MV3 service-worker termination Partly. The worker is reachable as a Playwright ServiceWorker and can be stopped, so "does state survive a worker restart" is testable. Genuine idle-timeout eviction on Chrome's real schedule is not something I can honestly claim to reproduce; the fix should be verified by forced termination plus code review, and I will say so rather than overclaim.
#153 Firefox target non-functional No - not by this harness. This is the honest limit. The whole premise of #153 is Chrome callback APIs used against Firefox's promise-only browser namespace. Verifying that fix in Chrome would verify nothing at all. It needs Firefox specifically, and Playwright's public API does not install extensions in Firefox. I have a separate probe running on the geckodriver / web-ext route and will report its real result, positive or negative.
A real dApp with real funds Still human. Signing against a live dApp with actual value is not something to simulate loosely, and I am not going to pretend a harness discharges it.

#171 specifically

The open question on PR #171 was an unverified Chrome pass on the dApp approval
signing path. That is within reach of this harness rather than needing your
time: a local test page that speaks EIP-1193 to window.ethereum can drive
eth_sendTransaction, personal_sign and eth_signTypedData_v4 through the
real content script, the real background worker and the real approval popup,
with the RPC stubbed. That is worth more than any single issue on the milestone
and I am sequencing it directly after the harness lands.

It does not cover the "real dApp, real funds" case, which stays yours.

Re-scoping this issue

Option (b) in the original body was costed as "a large detour". That costing was
wrong, because it was built on the same bad assumption as the rest of the issue.
The harness is a pinned image plus one dev dependency, it stays out of make test (the 20-second cap in REPO_POLICIES is real), and it earns its place
immediately by catching a live bug on day one.

So I am proceeding with (b), scoped to Chrome, filed as its own issue with
a definition of done that requires demonstrating a caught failure. Firefox is
tracked separately pending the probe result. Nothing on the milestone stays
paused waiting on this decision - the three options in the original body are
withdrawn, because they were a choice between bad answers to a question that
turned out not to exist.

The only thing genuinely reserved for you is the real-dApp-real-funds pass
before the 1.0.0 tag. That is a much smaller ask than this issue originally
made of you.

## Correction: the central premise of this issue is false I wrote this issue from an inference about the environment rather than a measurement of it. I have now measured it. **There is a browser available, it runs in a container, and it can drive this extension.** Everything below is observed output, not a plan. ### What I actually ran Image, pinned by digest: ``` mcr.microsoft.com/playwright@sha256:35246d87a7c88ea9b771c65d33171b2611b02a8253b4b12ce6f94376c55f99f2 ``` with `playwright-core@1.56.0`, launching the unpacked Chrome MV3 build: ```js chromium.launchPersistentContext(userDir, { channel: "chromium", // NOT the default headless shell - that one headless: true, // silently refuses to load extensions args: [ `--disable-extensions-except=${extPath}`, `--load-extension=${extPath}`, "--no-sandbox", ], }); ``` Result on the first run against `dist/chrome/`: ``` EXTENSION_ID=cieocojfinnamfiijllmlebfjdkedmfp POPUP_TITLE=AutistMask VISIBLE_VIEWS=["view-welcome"] BODY_TEXT_SNIPPET="DEBUG / INSECURE (welcome)\nAutistMask by @sneak\n..." ``` The service worker starts, the extension id is discoverable from its URL, and `chrome-extension://<id>/src/popup/index.html` renders and is fully scriptable. ### It survives a real end-to-end flow Driving the actual UI - `#btn-welcome-add`, `#btn-generate-phrase`, fill password, `#btn-add-wallet-confirm` - produces a real wallet and lands on `#view-main`. That exercises BIP-39 generation, libsodium vault encryption and extension storage persistence for real, in a real browser, not in jsdom. ### The part that matters: I proved the harness can FAIL A harness that has never been shown to fail is worth nothing, and this repo has already shipped two vacuous checks that passed on things they could not detect. So I pointed it at a bug that is **live on `main` right now** - #150, where `src/popup/views/addToken.js` line 24 calls `showView(...)` without importing it. I did not have to plant anything. ``` STEP wallet-created OK STEP address-detail OK CHECK_150 addTokenVisible=false errors=["pageerror: showView is not defined"] ``` The harness caught it: the screen did not open, and the exact `ReferenceError` was captured. `make check` is green on this same tree. That is the entire argument for the harness in one line - **`make check` passes, the screen is broken, and only the browser knows.** ### It also found a bug nobody had reported Every popup load emits an uncaught error: ``` pageerror: Aborted(CompileError: WebAssembly.instantiate(): Refused to compile or instantiate WebAssembly module because neither 'wasm-eval' nor 'unsafe-eval' is an allowed source of script in the following Content Security Policy directive: "script-src 'self'") ``` libsodium's WASM build is refused by the extension CSP and falls back to asm.js. Wallet creation still works, so this is not fatal, but a wallet's crypto library silently failing its preferred backend on every single load is not something to discover at 1.0. Filed separately. Also observed: the popup makes outbound API calls that returned `401` in the container. The harness must intercept network at the browser level so it is deterministic and offline, rather than depending on third-party endpoints. ### Corrected coverage assessment Reassessing the exposure list from the issue body against what was measured, rather than against what I assumed: | Item | Verifiable by the Chrome harness? | | --- | --- | | #150 AddToken unreachable | **Yes - already demonstrated above.** | | #151 TransactionDetail crash | **Yes** - same mechanism; needs a stubbed Blockscout response to seed an ERC-20 tx, which is exactly why network interception belongs in the harness. | | #155 WaitTx 60s timeout overwrites success | **Yes** - clock and network are both controllable from the harness; this becomes a deterministic test rather than a 60-second wait. | | #161 show recovery phrase | **Yes** - password-gated popup flow, fully drivable. | | #162 delete an address from an HD wallet | **Yes** - same. | | #158 MV3 service-worker termination | **Partly.** The worker is reachable as a Playwright `ServiceWorker` and can be stopped, so "does state survive a worker restart" is testable. Genuine idle-timeout eviction on Chrome's real schedule is not something I can honestly claim to reproduce; the fix should be verified by forced termination plus code review, and I will say so rather than overclaim. | | #153 Firefox target non-functional | **No - not by this harness.** This is the honest limit. The whole premise of #153 is Chrome callback APIs used against Firefox's promise-only `browser` namespace. Verifying that fix in Chrome would verify nothing at all. It needs Firefox specifically, and Playwright's public API does not install extensions in Firefox. I have a separate probe running on the geckodriver / `web-ext` route and will report its real result, positive or negative. | | A real dApp with real funds | **Still human.** Signing against a live dApp with actual value is not something to simulate loosely, and I am not going to pretend a harness discharges it. | ### #171 specifically The open question on PR #171 was an unverified Chrome pass on the dApp approval signing path. That is **within reach of this harness** rather than needing your time: a local test page that speaks EIP-1193 to `window.ethereum` can drive `eth_sendTransaction`, `personal_sign` and `eth_signTypedData_v4` through the real content script, the real background worker and the real approval popup, with the RPC stubbed. That is worth more than any single issue on the milestone and I am sequencing it directly after the harness lands. It does not cover the "real dApp, real funds" case, which stays yours. ### Re-scoping this issue Option (b) in the original body was costed as "a large detour". That costing was wrong, because it was built on the same bad assumption as the rest of the issue. The harness is a pinned image plus one dev dependency, it stays out of `make test` (the 20-second cap in REPO_POLICIES is real), and it earns its place immediately by catching a live bug on day one. So I am proceeding with **(b)**, scoped to Chrome, filed as its own issue with a definition of done that requires demonstrating a caught failure. Firefox is tracked separately pending the probe result. Nothing on the milestone stays paused waiting on this decision - the three options in the original body are withdrawn, because they were a choice between bad answers to a question that turned out not to exist. The only thing genuinely reserved for you is the real-dApp-real-funds pass before the 1.0.0 tag. That is a much smaller ask than this issue originally made of you.
clawbot commented 2026-08-09 16:28:45 +02:00
Author
Collaborator
Copy Link

Firefox probe result, closing the one open row in the coverage table above.

I said I would report it either way. It came back positive: automated Firefox
extension verification works today, not "achievable with modest work".

geckodriver 0.36.0 + stock Firefox 153.0.3, headless in a 259 MB container with
zero npm dependencies - the client is plain Node against geckodriver's HTTP
API. The MV2 build installs from dist/firefox/ as a temporary add-on with no
signing, and the popup uuid is pinned via a profile pref rather than discovered.
It was demonstrated failing on the live
#150 bug and passing on a fixed
build - different exit codes on identical inputs modulo one line.

Full specification, including the digests to pin and two traps that will
silently produce a vacuous harness if missed, is in
#184.

So the corrected row is:

Item Verifiable?
#153 Firefox target Yes - by the Firefox harness in #184, not by Chrome.

One correction to my own summary, in the interest of not overstating things.
The probe also observed that the Firefox popup works fine - wallet creation and
all - and suggested that undermines
#153. It does not, and I checked
before repeating it. The popup path uses the promise form of the storage API
and is correct on Firefox; the call sites that issue is about are in the content
script and background page, use the callback form, and the probe never walked
them. Reasoning recorded at
#153 (comment).

That distinction is the whole point of this exercise: an observation about paths
a probe did not execute is not evidence, and treating it as evidence is how this
issue got written on a false premise in the first place.

Net effect on the 1.0.0 milestone. Of the eight items originally listed here
as un-verifiable, seven now have a machine path: 150, 151, 155, 161, 162 and 153
fully, 158 partly (forced worker termination yes, real idle eviction no). The
only thing still genuinely reserved for you is a real dApp with real funds
before the tag.

This issue has nothing left to decide and I would close it once the harnesses
land, keeping it as the record of how the assumption got made and corrected. It
is unassigned from you as far as I am concerned - say the word if you would
rather it stay open.

Firefox probe result, closing the one open row in the coverage table above. I said I would report it either way. It came back positive: **automated Firefox extension verification works today**, not "achievable with modest work". geckodriver 0.36.0 + stock Firefox 153.0.3, headless in a 259 MB container with **zero npm dependencies** - the client is plain Node against geckodriver's HTTP API. The MV2 build installs from `dist/firefox/` as a temporary add-on with no signing, and the popup uuid is pinned via a profile pref rather than discovered. It was demonstrated failing on the live https://git.eeqj.de/sneak/AutistMask/issues/150 bug and passing on a fixed build - different exit codes on identical inputs modulo one line. Full specification, including the digests to pin and two traps that will silently produce a vacuous harness if missed, is in https://git.eeqj.de/sneak/AutistMask/issues/184. So the corrected row is: | Item | Verifiable? | | --- | --- | | https://git.eeqj.de/sneak/AutistMask/issues/153 Firefox target | **Yes** - by the Firefox harness in https://git.eeqj.de/sneak/AutistMask/issues/184, not by Chrome. | One correction to my own summary, in the interest of not overstating things. The probe also observed that the Firefox popup works fine - wallet creation and all - and suggested that undermines https://git.eeqj.de/sneak/AutistMask/issues/153. **It does not**, and I checked before repeating it. The popup path uses the promise form of the storage API and is correct on Firefox; the call sites that issue is about are in the content script and background page, use the callback form, and the probe never walked them. Reasoning recorded at https://git.eeqj.de/sneak/AutistMask/issues/153#issuecomment-49869. That distinction is the whole point of this exercise: an observation about paths a probe did not execute is not evidence, and treating it as evidence is how this issue got written on a false premise in the first place. **Net effect on the 1.0.0 milestone.** Of the eight items originally listed here as un-verifiable, seven now have a machine path: 150, 151, 155, 161, 162 and 153 fully, 158 partly (forced worker termination yes, real idle eviction no). The only thing still genuinely reserved for you is a real dApp with real funds before the tag. This issue has nothing left to decide and I would close it once the harnesses land, keeping it as the record of how the assumption got made and corrected. It is unassigned from you as far as I am concerned - say the word if you would rather it stay open.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
merge-ready
needs-checks
needs-rebase
needs-review
needs-rework
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
clawbot sneak
No Assignees sneak
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/AutistMask#173
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?