Uh oh!
There was an error while loading. Please reload this page.
feat(cu): complete verified background semantic execution - #1263
feat(cu): complete verified background semantic execution#1263hqhq1025 wants to merge 5 commits into
Conversation
0d71cba to
1a02146Compare1a02146 to
20990b4Compare
Astro-Han
left a comment
There was a problem hiding this comment.
One more P2 is not inline because the description line was unchanged: the model-facing tool text still says press_key is disabled by default, while this PR makes element-bound press_key a required successful action in the L2 scenario. Update the description and add a contract assertion.
The new scenario labels for focus and cursor safety and zoom crop also need evaluators. Today the validator checks the labels, not foreground pid, cursor movement, z-order, or crop coordinates.
| @@ -1973,16 +1893,28 @@ export function createCuaDriverBackend(opts: CuaDriverBackendOptions): CuDispatc | |||
| ); | |||
| if ('outcome' in validated) return validated; | |||
| if (action.type === 'press_key') { | |||
There was a problem hiding this comment.
[P1] This branch removed both physicalInputFailure() and the compatibility gate, but the pinned driver implements press_key by focusing the AX element and posting a keyboard CGEvent to the target pid. A user typing at the same time can still be interrupted or redirected. Restore the physical-input fence before dispatch; the current test even sets physicalInputRecentlyActive: () => true and expects success, so it locks in the unsafe behavior.
| tool: 'press_key', | ||
| pid: validated.pid, | ||
| windowId: validated.windowId, | ||
| address: 'ax', |
There was a problem hiding this comment.
[P1] address: 'ax' is not the driver's evidence. The pinned driver returns path: 'key_events', verified: false, effect: 'unverifiable' for press_key; hard-coding AX lets a CGEvent path satisfy the provider matrix's safe-dispatch gate. Derive the trace address from the normalized driver result and keep this path out of AX qualification.
| return deliveredVerificationFailure(action.type, 'ax'); | ||
| } | ||
| if (action.type === 'set_value') { | ||
| const exactIndexElement = fresh.elements.find( |
There was a problem hiding this comment.
[P2] element_index belongs to one snapshot and can be reassigned after the AX tree changes. Preferring the same numeric index in the fresh snapshot can verify a different element with the same role and label, even when the original target did not change. Match a stable identity, or require a unique role, label, and frame match and return outcome_unknown when identity cannot be proven.
| : matchingElements.length === 1 | ||
| ? matchingElements[0] | ||
| : undefined; | ||
| if (!readbackElement || readbackElement.value !== action.value) { |
There was a problem hiding this comment.
[P2] This equality makes valid writes unverifiable for supported inputs. Empty AX text values are omitted from the structured snapshot, and numeric slider or stepper values are not exposed as strings, so a successful clear or numeric set always becomes outcome_unknown. Use target-specific readback that distinguishes empty, numeric, and unreadable values, or reject roles that cannot be verified before delivery.
| { ...context, boundAction: boundElementAction(observation, '7') }, | ||
| ); | ||
| assert.equal(result.outcome.ok, true); |
There was a problem hiding this comment.
[P2] This test double reports evidence the pinned driver never returns for press_key: ax/verified/confirmed instead of key_events/unverifiable. It also never asserts the normalized tier or evidence, so the production contract can be wrong while this test stays green. Make the fixture return the pinned shape and assert the result and trace.
Review raised two P1s on this branch and both hold up against the driver's
actual behaviour.
**press_key is not an AX dispatch.** The trace hardcoded `address: 'ax'`
before the call even happened. Verified against cua-driver v0.12.6 with an
element-bound call, the driver returns:
{"effect": "unverifiable", "path": "key_events", "verified": false}
It resolves the element through AX and then posts keyboard CGEvents to the
pid. Labelling that as AX let a synthetic key path pass the safe-dispatch
gate that exists to keep pixel/event delivery out of AX qualification. The
trace address is now derived from the driver's own evidence, and the
delivered-but-unverified path reports `key_events` instead of `ax`.
**The physical-input fence still applies to it.** This branch removed the
fence for the whole semantic path. That is correct for `click_element` and
`set_value`, which are pure AX mutations, but not for `press_key`: keyboard
CGEvents interleave with — and can be redirected by — a user typing at the
same time. Element-bound addressing does not remove that hazard. The fence
is restored on this path only.
The test that pinned the old behaviour asserted success with
`physicalInputRecentlyActive: () => true`, welding the unsafe behaviour in
place. It now asserts the opposite, and a second test covers the trace
address. The test double also returned `path: 'ax', verified: true,
effect: 'confirmed'` for press_key — values the real driver never emits —
so it now mirrors the verified response.
Verified: `cua-driver-backend` suite 91 passed, 0 failed.
Summary
Rebase status
PR1 (#1262) merged on 2026-07-20. This branch has been rebased onto current
main, so the diff now contains only this PR's four commits.Because #1262 landed as a squash merge, a plain
git rebase mainwould have replayed PR1's commits against their own already-merged content. The rebase was therefore performed as:This applied cleanly with zero conflicts, and the resulting diff is byte-identical in scope to the pre-rebase delta (11 files, +552/-141).
Excluded scope
The old cursor-size experiments from the source worktree were deliberately excluded. Cursor visuals are handled separately by #1255.
Validation
Re-run after the rebase:
@maka/computer-use— 138/138 passed, including all sixcua-driversuites (backend, release contract, service lifecycle, snapshot coordinate authority, AX hit testing)npm run test:scripts— 121/121 passed, covering the four modified scripts@maka/runtime— 2503/2512 passed. The two failures (macOS filesystem worker smoke,builtin Bash streaming output) are load-sensitive timing tests in files this PR does not touch; both pass on re-run under normal load (55/55).Carried over from before the rebase (code unchanged by the rebase, so these remain applicable):
set_value→press_key→click_elementNot re-run after the rebase: the real-machine qualification above. CI is expected to cover the automated surface.