test(cu): drive the real desktop chain, and check the result from outside it by hqhq1025 · Pull Request #1886 · apache/maka

hqhq1025 · 2026-08-02T06:36:46Z

The gap these close

Everything before this drove the Computer Use backend from a bare node process:

node process ──→ backend ──→ executor ──→ macOS

That proves the protocol and the host. It says nothing about the chain a person uses:

composer ──→ tool-assembly ──→ backend ──→ executor ──→ macOS
↑ │
└──── cursor overlay ──────────┘
the mirror

Six defects lived in that gap — mock-green, CI-green, broken on a real machine.

Four scripts, four questions

script	question
`cu-desktop-chain-real.mjs`	does the whole chain work, from a prompt typed into the real composer?
`cu-desktop-matrix-real.mjs`	does it work for models other than the best one?
`cu-desktop-scenarios-real.mjs`	does it work for what people actually ask for?
`cu-overlay-repro.mjs`	can a presentation bug be reproduced without paying for a model turn?

The matrix is there because a tool surface only the strongest model can use is a tool surface designed badly. Weaker models hit the problems first and harder. Agreement across models means the surface is clear; a detour every model takes is the surface forcing it.

The scenarios are written from real requests — save this as a PDF, find the TODOs, move the window left — not from the capability list. Writing them from the capability list only proves the working things still work.

Target is Calculator: a real Apple application rather than a fixture, no documents, no network, no state worth keeping, and pressing its buttons is visible and reversible.

The part that matters most

Each script asserts the invariants a run must not break — the target never comes forward, the app keeps focus — rather than only that the model said it succeeded.

A model observes through the tool and reports through the same tool. On that one path, a click that never landed and a report saying it did are indistinguishable. Checking from outside is the point.

Verification

Four new files, no existing file touched. node --check on all four; the storage exports they use (createConnectionStore, createFileCredentialStore) resolve against current main.

…side it Everything before this drove the Computer Use backend from a bare node process. That proves the protocol and the host, and nothing about the chain a person actually uses: the renderer's composer, the tool surface assembled in tool-assembly.ts, the cursor overlay, the mirror. Those only exist inside the app, and six defects lived in that gap — mock-green, CI-green, broken on a real machine. Four scripts, each answering a different question. `cu-desktop-chain-real.mjs` launches the actual app with Playwright, types a prompt into the real composer, and drives a real Calculator. Calculator because it is a real Apple application rather than a fixture, has no documents, no network and no state worth keeping, and pressing its buttons is visible and reversible. `cu-desktop-matrix-real.mjs` puts the same tasks to several models across two families and three tiers. A tool surface only the best model can use is a tool surface designed badly, and the weaker models hit the problems first and harder. Agreement across models means the surface is clear; a detour every model takes is the surface forcing it. `cu-desktop-scenarios-real.mjs` asks for things people ask for — save this as a PDF, find the TODOs, move the window left — rather than for what the tool already supports. Writing the scenarios from the capability list only proves the working things still work; writing them from real requests is what finds the edges. `cu-overlay-repro.mjs` reproduces overlay-window behaviour without a model in the loop, so a presentation bug can be looked at without paying for a turn. Each asserts the invariants a run must not break — the target never comes forward, the app keeps focus — rather than only that the model said it succeeded. A model reports through the same path it observes through, so its own account of a run cannot distinguish a click that landed from one that did not.

Review found this harness could report success for a run that measured nothing, and that several of its world-checks passed without the model doing anything. The witness was not in this branch at all. Both files shell out to `scripts/cu-ax-oracle.swift`, which was added by the sibling evaluator pull request and exists on neither main nor here. So the oracle always threw, the catch turned it into `{error:'oracle_failed'}`, every scenario skipped, and the run exited 0 printing "0/0 checks passed". The independent-witness premise of the whole change was absent from it. The oracle moves here, where its callers are, rather than the two branches being stacked — it had no consumer on the other one, and stacking would have blocked this on an unrelated dependency. Zero checks is now a failure, and so is a skipped scenario or a harness crash. `report.flatMap(r => r.checks ?? [])` erased both, and `failed.length === 0` over an empty array is success. Three world-checks passed on state the model never produced. `fontbook-search` matched /courier/i over the whole Font Book tree, and Courier ships with macOS. `dictionary-lookup` had the same shape, and Dictionary restores its last lookup on launch, so a rerun passed on the previous run's state. `calculator-menu` compared button count and window width against fixed thresholds with no reference to `before`, so a Calculator left scientific by the best-effort teardown passed forever. All three now get what `textedit-typing` already had: a pre-check that the state was not there to begin with, and for the two search tasks, the query read out of the search field rather than out of the window at large. `dictionary-scroll` computed `start` and used it only in the detail string. A window restored already scrolled passed with no action, and so did a scroll that went upwards. It compares against `start` now. The anti-bypass check searched `argsHead`, the first 200 characters. Any command with a preamble, a `cd` or a heredoc pushed the `osascript` past 200 and became invisible — on the one check that exists because a model was caught driving the app around Computer Use while the oracle reported a clean green. It searches the whole argument string and keeps only the excerpt. The chain harness graded "the turn produced a Computer Use tool call" by regexing /Maka Computer|computer/i over `[class*="turn"]`, which matches `section.maka-turn` — the element that renders the user's own text. Every prompt in these files begins 用 computer use, so the check passed on the echo of its own request: withhold the tool surface entirely, let the model answer in prose with no tool call, and it still printed PASS. The same inclusion meant the empty-text guard three lines down could never see an empty transcript. (`.maka-chat-turn`, the other half of that selector, matches nothing in the product.) Both now read the session record: a `tool_call` message is one the model made, and the breakdown guard reads the assistant's text alone. `frontmostPid()` returned the string 'unavailable' on any osascript failure, and both background invariants are `!== targetPid`, so they held unconditionally. The invariant the product rests on — driving in the background never brings the target forward — passed exactly when the harness had lost the ability to check it, which an Automation permission denial does silently. An unreadable probe is now a failure. The scenarios runner destructured the child's exit code and never looked at it. The child exits 2 before asking the model anything when the screen is locked, when the target did not start, or when Computer Use could not be enabled; those runs recorded no checks, ran the witness against untouched state, and printed "not done" — the same cell a model that tried and failed produces. They print "did not run" now and are excluded from the count. `find-in-big-window` asserts that a read-only task changed nothing, so it can never be "not done", and it occupied a permanent success cell in the one number the file is read for. Read-only scenarios are scored "intact" and left out of that count. The call-count column was scraped from 操作电脑 N 次, a string that lives in unmerged interface work and appears nowhere in the product on main. Every cell was a constant 0 dressed as a measurement. The child knows the number from the session record and prints it; the parent reads that. The oracle itself gains two things. It never checked `AXIsProcessTrusted()`: without Accessibility permission every attribute read fails, `windows` is empty, and it emitted a well-formed `{"window_count":0,"element_count":0}` with no error — a blindfolded witness reporting an empty world, which given this project's history with TCC and node processes is the likeliest way it lies. And hitting `--depth` pruned the subtree while reporting `"truncated": false`, so every negative assertion made off it — the sidebar item is gone, one of the two windows is gone — could be reading a fact about the walk. Both are reported, and the callers that make negative assertions now refuse to draw one from a tree that was cut short. A witness that returns an error is raised rather than handed back as an empty tree. `MAKA_CU_DEBUG_LOG` is still set here and nothing on main writes it, so those traces come out empty until the executor branch lands. That is stated at the point where it is set, and the analyser on the other branch exits 2 on an empty corpus rather than reporting a clean run over nothing.

hqhq1025 · 2026-08-03T07:02:40Z

All nine findings confirmed against the code and fixed. Every new assertion was negative-controlled by constructing the world in which nothing happened and checking that the harness now says so; the before and after outputs are below.

On the dependency ordering

The witness was genuinely absent from this branch. cu-desktop-matrix-real.mjs and cu-desktop-scenarios-real.mjs shell out to scripts/cu-ax-oracle.swift, which was added by pr/cu-eval-harness and existed on neither main nor here, so the oracle always threw, the catch turned it into oracle_failed, every scenario skipped, and the run exited 0.

I moved the oracle here rather than stacking the branches. It had no consumer on the evaluator branch, and stacking would have blocked this pull request on that branch's own unresolvable dependency — its evaluator imports a renderer module that exists on no open pull request. Moving it makes both branches self-contained. The two oracle findings are fixed here as part of that move.

MAKA_CU_DEBUG_LOG is still set here and nothing on main writes it, so those traces come out empty until the executor branch lands. That is now stated at the point where the variable is set, and the analyser on the other branch exits 2 on an empty corpus rather than reporting a clean run over nothing.

Finding 1, the exit code

Confirmed. Controlled against the file's own summary block, run over synthetic reports.

Before, every scenario skipped: "0/0 checks passed", exit 0.
After, every scenario skipped: two SKIP lines, "2 of 2 scenario(s) never produced a check. That is a failed run, not a clean one.", "no check ran at all", exit 1.
After, harness crashed following one good scenario: CRASH line, exit 1.
After, one check failed: exit 1. After, everything passed: exit 0.

Finding 2, the tool-call check

Confirmed. .maka-chat-turn matches nothing in the product; the class is maka-turn. [class*="turn"] matches section.maka-turn, which renders turn.user.text.

Control: a turn with the tool surface withheld, the model answering in prose, zero tool calls.
Before: PASS, because the haystack contained the prompt's own 用 computer use. The empty-text guard saw 78 characters, so it too could never fire.
After: FAIL, 0 of 0 tool calls, and the breakdown guard reads only the assistant's text. On a turn that did call the tool: PASS.

Finding 3, the call count

Confirmed. 操作电脑 N 次 appears nowhere in the product; CU calls render as 调用 N 个工具 in packages/ui/src/tool-activity/copy.ts. Fixed at the source rather than removed: the child knows the count from the session record and prints one machine-readable line, and the parent reads that. Before, on a real child transcript: []. After: [[2, 0]].

Finding 4, three checks that pass on nothing

Confirmed for all three, though with one correction on the reasoning for Font Book: on my machine Courier was not in the tree on a fresh launch, because the list opens at the top and is virtualised. The defect is real regardless — the check was state-dependent and could go green without the model acting — and it gets the same treatment either way.

Controls, worlds in which the model did nothing:

fontbook-search with Courier already listed and an empty search field. Before: PASS. After: three FAILs. On a real search: three PASSes.
dictionary-lookup with the word restored from a previous run. Before: PASS. After: FAIL on both the pre-check and the effect. On a real lookup: PASS.
calculator-menu already scientific from a missed teardown. Before: PASS, "30 buttons, window width 600". After: FAIL, "the calculator started in the basic layout". On a real switch: PASS, "19 to 30 buttons, width 250 to 600".

Finding 5, dictionary-scroll

Confirmed. Before, restored already scrolled and nothing moved: PASS. Before, scrolled backwards 0.6 to 0.2: PASS. After: FAIL in both cases, PASS on 0 to 0.6.

Finding 6, the anti-bypass search

Confirmed. Control: a bash command with a preamble that puts osascript at index 261 of a 355-character argument string. Before, searching the first 200: MISSED. After: CAUGHT at 261. A benign command still reads clean. The excerpt is kept for the report, so nothing is truncated except what is printed.

Finding 7, the child's exit code

Confirmed. The code was destructured, stored, and never asserted.
Before, on a run where the child exited 2: cell reads "—", counted in the denominator of "attempts actually got the task done" as an ordinary miss.
After: "did not run (2)", the witness is not run at all because it would be reporting on the setup rather than on the attempt, and a line names it: "1 attempt(s) never ran ... Those say nothing about any model."

Finding 8, find-in-big-window

Confirmed. Read-only scenarios are scored "intact" or "DISTURBED" and excluded from the completion count. Before, on the same three results: 1/3, where the 1 was this free point. After: 0/1 scored, with the read-only attempt reported separately.

Finding 9, frontmostPid

Confirmed. Control: the probe's own source run with an exec that raises the Apple-events -1743 denial.
Before: returned "unavailable", and "driving in the background never brought the target forward" reported PASS.
After: returns null, and the check reports FAIL with "the frontmost process could not be read".

The oracle, findings 11 and 12

Depth truncation. Control on a real Finder tree at --depth 1, which drops 26 of 28 elements. Before: truncated false. After: truncated true, truncated_by_depth true, deepest_cut 2. At depth 2 and above on the same tree, truncated false — so the flag is not simply pinned on.

not_permitted. AXIsProcessTrusted was never checked. Added, and controlled by inverting the condition, which emits the not_permitted object; restored, and the oracle reads normally. The consumers were fixed alongside: the scenarios oracle now raises on any error field instead of handing back an object whose missing elements array every caller read as "the thing is not there", and the two negative assertions that could be reading a depth cut — the sidebar item is gone, one of the two windows is gone — refuse to draw a conclusion from a tree that was cut short. The matrix gained one check for the same reason, since all of its new pre-checks are negative assertions about the before tree.

Note on what could not be run

The harnesses themselves need a built desktop app and a live model, and this machine cannot build the workspace (node 22.11 here, no node:sqlite). The controls above therefore drive the shipped code by slicing the functions and blocks under test verbatim out of the files rather than by reimplementing them, so a later change to the logic takes the control with it. npx biome check is clean on all four files and node scripts/check-console.mjs passes.

Nobody reviewing this could run it. All four scripts need a real desktop session, a real model connection and the reviewer's own provider credentials, so the only person able to judge whether they work is the person who wrote them. That is not a check; it is a claim. And the claim was wrong. Measured on this branch: `cu-desktop-scenarios-real.mjs` never exits non-zero. The only `process.exit` is the 2 at line 581 for an unknown scenario key, and the file ends on a `console.log`. Delete Computer Use from the product entirely and it prints `0/13 attempts actually got the task done` and exits 0. Both Electron drivers die before they send a prompt. `chain-real.mjs:199,231,232` and `matrix-real.mjs:441` wait on `.maka-composer-textarea`, which has zero occurrences in this repository — the composer is a contenteditable, and apps/desktop/e2e/fixtures.ts:21 has said so all along. That crash is then graded as a run: `scenarios-real.mjs:677` reads `code === 0 || code === 1` as "the model was asked", and a selector timeout leaves `chain-real` exiting 1 through its own top-level catch. So the three defects compose into a harness that has never once driven the product and reports that it did. The five-model matrix ran one model. `seedConnection` writes a scratch connection into the temp user-data dir and then `chain-real.mjs:174-183` copies the developer's real `llm-connections.json`, `credentials.json` and `settings.json` over the same paths before launch, so all five columns are the developer's default connection under five different labels and nothing checks which model answered. `matrix-real.mjs:431` launches with no `--user-data-dir` at all, which writes sessions into the real Maka workspace. Smaller, same shape: the trace is computed, removed, passed as `MAKA_CU_DEBUG_LOG` and never read; the four PiP mirror checks sit inside `if (pip)` and no PiP window exists on main; `cu-overlay-repro.mjs:34` imports `main/computer-use/pip-window.js`, which is not in the built tree; `matrix-real.mjs:447` reads `settings.computerUse.enabled`, and `computerUse` is not a key in `AppSettings`; `find-todo` passes on the directory name, because the project is created at `find-todo-project` and the oracle records the window title, so a model that refuses outright scores `done`; and three guards are satisfied by their own failure values — a lock probe that reads a missing Swift toolchain as unlocked, a stop-button count that reads a Playwright error as "the turn finished", and a 240s expiry that runs the checks against a mid-flight transcript and records nothing. None of that is worth repairing here. These stay as local acceptance tools, run by hand against a real machine, where their cost is paid by the person who gets the benefit. The commit before this one keeps them intact. What was worth keeping is already kept. `cu-ax-oracle.swift` moves to the eval-harness branch with the analyser: it is deterministic, self-contained, and anyone on macOS can run it, which is the property every file above lacks.

hqhq1025 · 2026-08-03T08:36:10Z

This PR is now empty, and it should be closed rather than merged.

Everything it added has been withdrawn in 4dfafa4. The four real-desktop driver scripts — cu-desktop-chain-real.mjs, cu-desktop-matrix-real.mjs, cu-desktop-scenarios-real.mjs, cu-overlay-repro.mjs — stay as local acceptance tools, and cu-ax-oracle.swift has moved to #1884 so it ships with the trace analyser. git diff origin/main...HEAD is empty.

The scripts are preserved intact in the commit immediately before the removal: fcce03e. Nothing was lost.

The reason, since it is worth recording. The four scripts need a real desktop session, a real model connection and the author's own provider credentials, so nobody reviewing this PR could run them or judge whether they are correct. That alone would be a reason to keep them out of the repository. What settles it is that they were wrong in the direction that reads as success, and no reviewer could have found that either.

Every item below was measured against fcce03e, not inferred.

The scenario runner cannot fail. The only process.exit in cu-desktop-scenarios-real.mjs is the 2 at line 581 for an unknown scenario key; the file is 799 lines and ends on a console.log. Delete Computer Use from the product entirely and it prints "0/13 attempts actually got the task done" and exits 0.

Both Electron drivers die before they send a prompt. chain-real.mjs waits on .maka-composer-textarea at line 199 and fills it at 231 and 232; matrix-real.mjs waits on it at 441. That class has zero occurrences in apps and packages. The composer is a contenteditable, and apps/desktop/e2e/fixtures.ts:21 has said so all along: COMPOSER_INPUT = '.maka-composer-editor [contenteditable="true"]'.

That crash is then graded as a run. scenarios-real.mjs:677 reads const attempted = code === 0 || code === 1, and chain-real exits 1 both for a failed check and from its own top-level catch — which is what a selector timeout produces. Only exit 2 counts as "did not run". Taken together, this harness has never once driven the product, and it reported that it did.

The five-model matrix ran one model. chain-real.mjs:174-183 copies the developer's real llm-connections.json, credentials.json and settings.json from ~/Library/Application Support/Maka/workspaces/default over the seeded scratch profile before launch, so all five columns are the developer's default connection under five different labels, and nothing checks which model answered. Separately, matrix-real.mjs:431 is electron.launch({ args: ['.'], cwd: DESKTOP }) with no --user-data-dir at all, so it writes sessions into the real Maka workspace.

The trace is write-only. In scenarios-real.mjs, tracePath is computed at 587, removed at 594, passed as MAKA_CU_DEBUG_LOG at 620, and never read, stat'ed or parsed. Those are its only three occurrences.

cu-overlay-repro.mjs imports createComputerUsePipController from main/computer-use/pip-window.js. apps/desktop/src/main/computer-use/ contains one file, cursor-overlay-window.ts. There is no pip-window in the tree, built or otherwise, so the four PiP mirror checks could not have run even if the if (pip) gate had let them.

matrix-real.mjs:447 does settings.update({ computerUse: { enabled: true } }) and reads back .computerUse.enabled. AppSettings, at packages/core/src/settings.ts:266, has fifteen fields and computerUse is not one of them.

find-todo passes on the directory name: the project is created at find-todo-project (scenarios-real.mjs:203) and the oracle records the window's AXTitle, which contains "todo". A model that refuses outright scores done.

Three guards are satisfied by their own failure values. chain-real.mjs:130-140 runs the Swift lock probe through .catch(() => ({stdout:''})) and tests === 'locked', so a missing toolchain or a timeout reads as unlocked. Line 306 is stopButton.count().catch(() => 0) === 0, which reads a Playwright error as "the turn finished". Line 303 is a 240s expiry that then runs the checks against a mid-flight transcript with no record that it did.

What was good is kept. The Swift oracle works, and it is the one file here anyone on macOS can run and check; it is on #1884 now, with the depth-truncation and not_permitted fixes and a negative control for each. matrix-real.mjs:764-766 gating exit 0 on a non-empty check list, the truncated !== true gating on negative assertions, and the before/after differencing were all the right instincts, and they go with the scripts into fcce03e.

hqhq1025 · 2026-08-03T08:37:56Z

Closing this without merging. The four scripts in it drive a real desktop with a
real model connection and the author's own credentials, so nobody reviewing the
PR could run them or judge whether they were right — and a second review found
they had never successfully driven the product at all: they waited on a
.maka-composer-textarea selector that does not exist in this repository, died
before sending a prompt, and then graded that crash as "the model was asked",
with the scenario runner having no non-zero exit path at all. Delete Computer Use
from the product entirely and it printed "0 of 13 attempts actually got the task
done" and exited 0.

A harness whose silence reads as success is worse than no harness, so these stay
as local acceptance tools rather than repository code. They are preserved intact
at commit fcce03e on this branch.

The one piece worth sharing moved to #1884: scripts/cu-ax-oracle.swift, a
self-contained deterministic Accessibility probe anyone on macOS can run. It now
reports a depth cut as truncation instead of claiming the tree was complete, and
emits not_permitted with a non-zero exit when it has no Accessibility grant,
rather than printing a well-formed empty world that is indistinguishable from an
application with nothing in it.

hqhq1025 · 2026-08-02T06:36:46Z

The gap these close

Everything before this drove the Computer Use backend from a bare node process:

node process ──→ backend ──→ executor ──→ macOS

That proves the protocol and the host. It says nothing about the chain a person uses:

composer ──→ tool-assembly ──→ backend ──→ executor ──→ macOS
↑ │
└──── cursor overlay ──────────┘
the mirror

Six defects lived in that gap — mock-green, CI-green, broken on a real machine.

Four scripts, four questions

script	question
`cu-desktop-chain-real.mjs`	does the whole chain work, from a prompt typed into the real composer?
`cu-desktop-matrix-real.mjs`	does it work for models other than the best one?
`cu-desktop-scenarios-real.mjs`	does it work for what people actually ask for?
`cu-overlay-repro.mjs`	can a presentation bug be reproduced without paying for a model turn?

The matrix is there because a tool surface only the strongest model can use is a tool surface designed badly. Weaker models hit the problems first and harder. Agreement across models means the surface is clear; a detour every model takes is the surface forcing it.

The scenarios are written from real requests — save this as a PDF, find the TODOs, move the window left — not from the capability list. Writing them from the capability list only proves the working things still work.

Target is Calculator: a real Apple application rather than a fixture, no documents, no network, no state worth keeping, and pressing its buttons is visible and reversible.

The part that matters most

Each script asserts the invariants a run must not break — the target never comes forward, the app keeps focus — rather than only that the model said it succeeded.

A model observes through the tool and reports through the same tool. On that one path, a click that never landed and a report saying it did are indistinguishable. Checking from outside is the point.

Verification

Four new files, no existing file touched. node --check on all four; the storage exports they use (createConnectionStore, createFileCredentialStore) resolve against current main.

…side it Everything before this drove the Computer Use backend from a bare node process. That proves the protocol and the host, and nothing about the chain a person actually uses: the renderer's composer, the tool surface assembled in tool-assembly.ts, the cursor overlay, the mirror. Those only exist inside the app, and six defects lived in that gap — mock-green, CI-green, broken on a real machine. Four scripts, each answering a different question. `cu-desktop-chain-real.mjs` launches the actual app with Playwright, types a prompt into the real composer, and drives a real Calculator. Calculator because it is a real Apple application rather than a fixture, has no documents, no network and no state worth keeping, and pressing its buttons is visible and reversible. `cu-desktop-matrix-real.mjs` puts the same tasks to several models across two families and three tiers. A tool surface only the best model can use is a tool surface designed badly, and the weaker models hit the problems first and harder. Agreement across models means the surface is clear; a detour every model takes is the surface forcing it. `cu-desktop-scenarios-real.mjs` asks for things people ask for — save this as a PDF, find the TODOs, move the window left — rather than for what the tool already supports. Writing the scenarios from the capability list only proves the working things still work; writing them from real requests is what finds the edges. `cu-overlay-repro.mjs` reproduces overlay-window behaviour without a model in the loop, so a presentation bug can be looked at without paying for a turn. Each asserts the invariants a run must not break — the target never comes forward, the app keeps focus — rather than only that the model said it succeeded. A model reports through the same path it observes through, so its own account of a run cannot distinguish a click that landed from one that did not.

Review found this harness could report success for a run that measured nothing, and that several of its world-checks passed without the model doing anything. The witness was not in this branch at all. Both files shell out to `scripts/cu-ax-oracle.swift`, which was added by the sibling evaluator pull request and exists on neither main nor here. So the oracle always threw, the catch turned it into `{error:'oracle_failed'}`, every scenario skipped, and the run exited 0 printing "0/0 checks passed". The independent-witness premise of the whole change was absent from it. The oracle moves here, where its callers are, rather than the two branches being stacked — it had no consumer on the other one, and stacking would have blocked this on an unrelated dependency. Zero checks is now a failure, and so is a skipped scenario or a harness crash. `report.flatMap(r => r.checks ?? [])` erased both, and `failed.length === 0` over an empty array is success. Three world-checks passed on state the model never produced. `fontbook-search` matched /courier/i over the whole Font Book tree, and Courier ships with macOS. `dictionary-lookup` had the same shape, and Dictionary restores its last lookup on launch, so a rerun passed on the previous run's state. `calculator-menu` compared button count and window width against fixed thresholds with no reference to `before`, so a Calculator left scientific by the best-effort teardown passed forever. All three now get what `textedit-typing` already had: a pre-check that the state was not there to begin with, and for the two search tasks, the query read out of the search field rather than out of the window at large. `dictionary-scroll` computed `start` and used it only in the detail string. A window restored already scrolled passed with no action, and so did a scroll that went upwards. It compares against `start` now. The anti-bypass check searched `argsHead`, the first 200 characters. Any command with a preamble, a `cd` or a heredoc pushed the `osascript` past 200 and became invisible — on the one check that exists because a model was caught driving the app around Computer Use while the oracle reported a clean green. It searches the whole argument string and keeps only the excerpt. The chain harness graded "the turn produced a Computer Use tool call" by regexing /Maka Computer|computer/i over `[class*="turn"]`, which matches `section.maka-turn` — the element that renders the user's own text. Every prompt in these files begins 用 computer use, so the check passed on the echo of its own request: withhold the tool surface entirely, let the model answer in prose with no tool call, and it still printed PASS. The same inclusion meant the empty-text guard three lines down could never see an empty transcript. (`.maka-chat-turn`, the other half of that selector, matches nothing in the product.) Both now read the session record: a `tool_call` message is one the model made, and the breakdown guard reads the assistant's text alone. `frontmostPid()` returned the string 'unavailable' on any osascript failure, and both background invariants are `!== targetPid`, so they held unconditionally. The invariant the product rests on — driving in the background never brings the target forward — passed exactly when the harness had lost the ability to check it, which an Automation permission denial does silently. An unreadable probe is now a failure. The scenarios runner destructured the child's exit code and never looked at it. The child exits 2 before asking the model anything when the screen is locked, when the target did not start, or when Computer Use could not be enabled; those runs recorded no checks, ran the witness against untouched state, and printed "not done" — the same cell a model that tried and failed produces. They print "did not run" now and are excluded from the count. `find-in-big-window` asserts that a read-only task changed nothing, so it can never be "not done", and it occupied a permanent success cell in the one number the file is read for. Read-only scenarios are scored "intact" and left out of that count. The call-count column was scraped from 操作电脑 N 次, a string that lives in unmerged interface work and appears nowhere in the product on main. Every cell was a constant 0 dressed as a measurement. The child knows the number from the session record and prints it; the parent reads that. The oracle itself gains two things. It never checked `AXIsProcessTrusted()`: without Accessibility permission every attribute read fails, `windows` is empty, and it emitted a well-formed `{"window_count":0,"element_count":0}` with no error — a blindfolded witness reporting an empty world, which given this project's history with TCC and node processes is the likeliest way it lies. And hitting `--depth` pruned the subtree while reporting `"truncated": false`, so every negative assertion made off it — the sidebar item is gone, one of the two windows is gone — could be reading a fact about the walk. Both are reported, and the callers that make negative assertions now refuse to draw one from a tree that was cut short. A witness that returns an error is raised rather than handed back as an empty tree. `MAKA_CU_DEBUG_LOG` is still set here and nothing on main writes it, so those traces come out empty until the executor branch lands. That is stated at the point where it is set, and the analyser on the other branch exits 2 on an empty corpus rather than reporting a clean run over nothing.

hqhq1025 · 2026-08-03T07:02:40Z

All nine findings confirmed against the code and fixed. Every new assertion was negative-controlled by constructing the world in which nothing happened and checking that the harness now says so; the before and after outputs are below.

On the dependency ordering

The witness was genuinely absent from this branch. cu-desktop-matrix-real.mjs and cu-desktop-scenarios-real.mjs shell out to scripts/cu-ax-oracle.swift, which was added by pr/cu-eval-harness and existed on neither main nor here, so the oracle always threw, the catch turned it into oracle_failed, every scenario skipped, and the run exited 0.

I moved the oracle here rather than stacking the branches. It had no consumer on the evaluator branch, and stacking would have blocked this pull request on that branch's own unresolvable dependency — its evaluator imports a renderer module that exists on no open pull request. Moving it makes both branches self-contained. The two oracle findings are fixed here as part of that move.

MAKA_CU_DEBUG_LOG is still set here and nothing on main writes it, so those traces come out empty until the executor branch lands. That is now stated at the point where the variable is set, and the analyser on the other branch exits 2 on an empty corpus rather than reporting a clean run over nothing.

Finding 1, the exit code

Confirmed. Controlled against the file's own summary block, run over synthetic reports.

Before, every scenario skipped: "0/0 checks passed", exit 0.
After, every scenario skipped: two SKIP lines, "2 of 2 scenario(s) never produced a check. That is a failed run, not a clean one.", "no check ran at all", exit 1.
After, harness crashed following one good scenario: CRASH line, exit 1.
After, one check failed: exit 1. After, everything passed: exit 0.

Finding 2, the tool-call check

Confirmed. .maka-chat-turn matches nothing in the product; the class is maka-turn. [class*="turn"] matches section.maka-turn, which renders turn.user.text.

Control: a turn with the tool surface withheld, the model answering in prose, zero tool calls.
Before: PASS, because the haystack contained the prompt's own 用 computer use. The empty-text guard saw 78 characters, so it too could never fire.
After: FAIL, 0 of 0 tool calls, and the breakdown guard reads only the assistant's text. On a turn that did call the tool: PASS.

Finding 3, the call count

Confirmed. 操作电脑 N 次 appears nowhere in the product; CU calls render as 调用 N 个工具 in packages/ui/src/tool-activity/copy.ts. Fixed at the source rather than removed: the child knows the count from the session record and prints one machine-readable line, and the parent reads that. Before, on a real child transcript: []. After: [[2, 0]].

Finding 4, three checks that pass on nothing

Confirmed for all three, though with one correction on the reasoning for Font Book: on my machine Courier was not in the tree on a fresh launch, because the list opens at the top and is virtualised. The defect is real regardless — the check was state-dependent and could go green without the model acting — and it gets the same treatment either way.

Controls, worlds in which the model did nothing:

fontbook-search with Courier already listed and an empty search field. Before: PASS. After: three FAILs. On a real search: three PASSes.
dictionary-lookup with the word restored from a previous run. Before: PASS. After: FAIL on both the pre-check and the effect. On a real lookup: PASS.
calculator-menu already scientific from a missed teardown. Before: PASS, "30 buttons, window width 600". After: FAIL, "the calculator started in the basic layout". On a real switch: PASS, "19 to 30 buttons, width 250 to 600".

Finding 5, dictionary-scroll

Confirmed. Before, restored already scrolled and nothing moved: PASS. Before, scrolled backwards 0.6 to 0.2: PASS. After: FAIL in both cases, PASS on 0 to 0.6.

Finding 6, the anti-bypass search

Confirmed. Control: a bash command with a preamble that puts osascript at index 261 of a 355-character argument string. Before, searching the first 200: MISSED. After: CAUGHT at 261. A benign command still reads clean. The excerpt is kept for the report, so nothing is truncated except what is printed.

Finding 7, the child's exit code

Confirmed. The code was destructured, stored, and never asserted.
Before, on a run where the child exited 2: cell reads "—", counted in the denominator of "attempts actually got the task done" as an ordinary miss.
After: "did not run (2)", the witness is not run at all because it would be reporting on the setup rather than on the attempt, and a line names it: "1 attempt(s) never ran ... Those say nothing about any model."

Finding 8, find-in-big-window

Confirmed. Read-only scenarios are scored "intact" or "DISTURBED" and excluded from the completion count. Before, on the same three results: 1/3, where the 1 was this free point. After: 0/1 scored, with the read-only attempt reported separately.

Finding 9, frontmostPid

Confirmed. Control: the probe's own source run with an exec that raises the Apple-events -1743 denial.
Before: returned "unavailable", and "driving in the background never brought the target forward" reported PASS.
After: returns null, and the check reports FAIL with "the frontmost process could not be read".

The oracle, findings 11 and 12

Depth truncation. Control on a real Finder tree at --depth 1, which drops 26 of 28 elements. Before: truncated false. After: truncated true, truncated_by_depth true, deepest_cut 2. At depth 2 and above on the same tree, truncated false — so the flag is not simply pinned on.

not_permitted. AXIsProcessTrusted was never checked. Added, and controlled by inverting the condition, which emits the not_permitted object; restored, and the oracle reads normally. The consumers were fixed alongside: the scenarios oracle now raises on any error field instead of handing back an object whose missing elements array every caller read as "the thing is not there", and the two negative assertions that could be reading a depth cut — the sidebar item is gone, one of the two windows is gone — refuse to draw a conclusion from a tree that was cut short. The matrix gained one check for the same reason, since all of its new pre-checks are negative assertions about the before tree.

Note on what could not be run

The harnesses themselves need a built desktop app and a live model, and this machine cannot build the workspace (node 22.11 here, no node:sqlite). The controls above therefore drive the shipped code by slicing the functions and blocks under test verbatim out of the files rather than by reimplementing them, so a later change to the logic takes the control with it. npx biome check is clean on all four files and node scripts/check-console.mjs passes.

Nobody reviewing this could run it. All four scripts need a real desktop session, a real model connection and the reviewer's own provider credentials, so the only person able to judge whether they work is the person who wrote them. That is not a check; it is a claim. And the claim was wrong. Measured on this branch: `cu-desktop-scenarios-real.mjs` never exits non-zero. The only `process.exit` is the 2 at line 581 for an unknown scenario key, and the file ends on a `console.log`. Delete Computer Use from the product entirely and it prints `0/13 attempts actually got the task done` and exits 0. Both Electron drivers die before they send a prompt. `chain-real.mjs:199,231,232` and `matrix-real.mjs:441` wait on `.maka-composer-textarea`, which has zero occurrences in this repository — the composer is a contenteditable, and apps/desktop/e2e/fixtures.ts:21 has said so all along. That crash is then graded as a run: `scenarios-real.mjs:677` reads `code === 0 || code === 1` as "the model was asked", and a selector timeout leaves `chain-real` exiting 1 through its own top-level catch. So the three defects compose into a harness that has never once driven the product and reports that it did. The five-model matrix ran one model. `seedConnection` writes a scratch connection into the temp user-data dir and then `chain-real.mjs:174-183` copies the developer's real `llm-connections.json`, `credentials.json` and `settings.json` over the same paths before launch, so all five columns are the developer's default connection under five different labels and nothing checks which model answered. `matrix-real.mjs:431` launches with no `--user-data-dir` at all, which writes sessions into the real Maka workspace. Smaller, same shape: the trace is computed, removed, passed as `MAKA_CU_DEBUG_LOG` and never read; the four PiP mirror checks sit inside `if (pip)` and no PiP window exists on main; `cu-overlay-repro.mjs:34` imports `main/computer-use/pip-window.js`, which is not in the built tree; `matrix-real.mjs:447` reads `settings.computerUse.enabled`, and `computerUse` is not a key in `AppSettings`; `find-todo` passes on the directory name, because the project is created at `find-todo-project` and the oracle records the window title, so a model that refuses outright scores `done`; and three guards are satisfied by their own failure values — a lock probe that reads a missing Swift toolchain as unlocked, a stop-button count that reads a Playwright error as "the turn finished", and a 240s expiry that runs the checks against a mid-flight transcript and records nothing. None of that is worth repairing here. These stay as local acceptance tools, run by hand against a real machine, where their cost is paid by the person who gets the benefit. The commit before this one keeps them intact. What was worth keeping is already kept. `cu-ax-oracle.swift` moves to the eval-harness branch with the analyser: it is deterministic, self-contained, and anyone on macOS can run it, which is the property every file above lacks.

hqhq1025 · 2026-08-03T08:36:10Z