Skip to content

fix(cu): let a model read back the Computer Use call it actually made - #1891

Merged
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types
Aug 3, 2026
Merged

fix(cu): let a model read back the Computer Use call it actually made#1891
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types

Conversation

@hqhq1025

@hqhq1025hqhq1025 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The record a model reads back of its own Computer Use call was the host's
approval projection, not the call. model-history.ts replays
event.content.args, and those args came from computerUseApprovalSummary,
which exists to decide and display a permission: it renames window_id to
windowId, adds approvalClass and rememberForTurnAllowed, and drops every
argument it does not need.

Measured on the real ToolRuntime, a model that sent

{"action":"press_key","app":"TextEdit","window_id":41,
"observation_id":"obs-7","element_id":"e12","text":"cmd+s"}

read back

{"action":"press_key","approvalClass":"keyboard_mutation",
"rememberForTurnAllowed":true,"app":"TextEdit","windowId":41,
"observationId":"obs-7"}

— a key the tool rejects, two fields it never sent, no element, and a
press_key with no key. A model imitates the shape it is shown, so it went on
calling it that way; a real desktop run failed six of eleven calls on shapes
copied from its own history.

This adds computerUseModelCallArgs: the same privacy rule, in the names the
tool accepts, and wires it at tool-runtime.ts, which is the line every
shipping backend goes through. The permission prompt still reads
permissionArgs and the approval scope key is still computed from the raw
call, so only what is written down changes.

What now crosses the privacy boundary

computer-use-privacy-boundary.test.ts asserted this line before this PR, and
it is updated in place rather than around, so the change is visible in the
diff. Two things cross that did not:

  • element_id — an index the model chose into an observation it had already
    read (e12). Admitted only when it is a stable identifier, so an
    accessibility label arriving under that key is dropped rather than persisted.
    Without it, ten element actions in a turn are indistinguishable.
  • coordinate, start_coordinate, region — geometry the model wrote into
    the call itself. Integers only, so a mistyped value still degrades to a
    shape. A coordinate is not read off the screen; withholding it leaves a model
    that clicked a point and missed unable to see that it had already tried that
    point.

What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed. Those keep their key and
lose their value to <text> — the key has to stay, or the model reads its own
history as a call it never made. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so it is
carried: one argument name meaning six things, and only two of the six come
from outside the model.

element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the model's
element_id against the live observation and attaches it, and permissionArgs
is what this projection runs on.

Also here

COMPUTER_USE_SEMANTIC_ACTIONS replaces a hand-written action list that had
already drifted: window_action was added to the strict union and not to the
list, so every window move, resize and minimise would have summarised as
unknown — in the approval a person reads, and in the record the model reads
back. computer-use-schema-parity.test.ts now compares the two in both
directions.

Relationship to #1888

#1888 reached the same conclusion from the renderer's side and carried
elementId in the approval summary so its labels could tell two element rows
apart. That intent is subsumed here: element_id is carried, in the tool's own
casing. When the two are merged, #1888's computer-action-label.ts should read
window_id and element_id rather than windowId and elementId, and the
elementId case it added to computer-use-privacy-boundary.test.ts is covered
by the projection assertion in this branch.

Not here any more

The executor capability declarations this PR was originally titled for were
removed in 0fcfb46: nothing produced them, so they were a type surface with
no writer. The title and this description are rewritten to match what the
branch actually does.

Verification

The new assertion is on the tool_start event, not on the projection function.
Reverted to computerUseApprovalSummary at tool-runtime.ts and rebuilt, it
fails with exactly the diff quoted above; restored, it passes. A test that
calls computerUseModelCallArgs directly stays green either way, which is how
this survived being fixed once already in PiAgentBackend — a backend the
desktop app never instantiates.

@maka/core and @maka/runtime suites pass; npx biome check clean on the
touched files; node scripts/check-console.mjs passes. Rebased onto current
main, cleanly.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all six findings. The claim in the PR body that nothing behaves differently was wrong, and finding 1 was the reason.

Finding 1, resolution: I scoped the constant to what main's schema accepts rather than stacking the PR on the executor branch. Verified first — main's wire enum at computer-use-tools.ts is list_apps, observe, click_element, set_value, select_text, secondary_action, press_key plus CU_ACTION_TYPES, and none of launch_app, scroll_element, element_sequence or window_action is in it. The four names went into APPROVAL_ACTIONS through COMPUTER_USE_SEMANTIC_ACTIONS, so computerUseApprovalSummary flipped knownAction to true for calls the SDK rejects before the tool runs, and rememberForTurnAllowed could evaluate true for the two that also joined SEMANTIC_ACTIONS. I chose scoping over stacking for two reasons. The constant's stated purpose is to be held against the tool schema, so declaring names the schema does not have is the same drift it exists to prevent, only in the direction nothing was checking. And stacking would make the PR unmergeable until the executor lands, which delays the parts that are correct today. The executor's vocabulary belongs on the branch that also ships the executor.

Finding 2: the promised file did not exist, but a guard did — computer-use-schema-parity.test.ts, added in #1870 after this branch was cut. It only walks from the schemas to the catalog, which catches a missing name and not an extra one, so the widening in finding 1 would have passed it. I added the reverse comparison to that file rather than creating the second one the comment named, and corrected the comment to point at the real file. Negative control: reintroducing the four names turns the new check red and it names all four. There is also an in-file control that runs the same comparator over a fixed catalog, so it keeps proving the comparator can return something on the day the real catalog is correct.

Finding 3: confirmed. Only COMPUTER_USE_SEMANTIC_ACTIONS was exported, so computerUseModelCallArgs was unreachable from @maka/runtime and the whole headline fix was dead code while the harmful widening was the only part that shipped. The five missing symbols are exported now, and the projection is wired into projectPiToolArgs in pi-agent-backend.ts, which writes the persisted tool_call message and the tool_start event. Negative control: reverting that one line turns the pi-agent-backend projection test red with the approval dialect back in the diff.

Finding 4: confirmed and dropped rather than wired. ComputerUseSettings appears in none of AppSettings, UpdateAppSettingsInput, the defaults, the merge or the normalize pass, unlike every peer settings module, and select-backend.ts assembles the CU tools with no enable check anywhere. A type that says enabled: false while the shipped app always has it on is worse than no type. Wiring a real switch needs a UI and a migration and does not belong under a types commit.

Finding 5: confirmed, and it is exactly as described. text is six arguments under one name — the key for press_key, key and hold_key, the element action name for secondary_action, a screen substring for select_text, typed content for type. MODEL_CALL_PLAIN_VALUES is now a map from action to the argument names that are plain for that action. An action the tool does not accept has no entry, so it falls through to shapes rather than to whichever set was used last. Negative control: reverting to the name-keyed set turns two tests red, showing press_key coming back as text: .

Finding 6, partially rejected with evidence. dispatch_refused is gone. It is not a declaration — COMPUTER_USE_ERROR_CODES is a runtime value feeding isComputerUseErrorCode, which decides at cua-driver-result.ts:88 whether a raw string from the driver process is trusted or replaced with capture_failed. Widening that with no producer is a live change. The other five named items are different in kind. messageIsAppTextFree, truncated, actions, subrole and placeholder are optional fields on TypeScript interfaces with no runtime effect, and they are not distinguishable from about ten other unproduced declarations in the same diff that the review accepted — focused, appAlias, obscuringRects, menu, query, CuLaunchedApp, scroll_element, window_action, launchApp. Dropping the named five and keeping those would be arbitrary; dropping all of them empties the branch of its purpose. Declaring the shape an executor returns is what this branch is for, so they stay, and the absent-means-withheld polarity on messageIsAppTextFree is kept as suggested.

Also rebased onto current main, which the branch was one commit behind — the missing commit was #1870, the parity test finding 2 turns out to depend on.

Test delta, measured against a baseline taken on clean origin/main on the same machine. This machine's Node has no node:sqlite, so @maka/runtime carries 64 pre-existing failing test names there. After the change: 2722 tests, zero new failures and zero fixed against that set. @maka/core goes 703/703 to 710/710, the seven new ones being the model-call projection tests. biome check clean on the six touched files, check-console passes.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Second-round review addressed. Two findings fixed, one dropped after checking, one scope decision.

Coordinates are back. The doc said typed text, written values and coordinates are screen-derived and stay out, and that is false for the third one: a coordinate is the model's own output, four digits it chose and sent. Reduced to <point>, a model that clicked [412, 88] and missed could not tell whether it had already tried that point, which is the repeated-call shape this projection exists to make visible. coordinate, start_coordinate and region now come back whole when they are integers and still degrade to a shape when they are not, so a malformed argument does not become a leak channel.

The out-of-enum action is reported as sent. On the AI SDK path the question is moot — a schema-rejected call never produces a tool-call part, so settleToolCall is never reached and no tool_start is emitted. But this projection's only consumer is the Pi backend, where tool_start arrives as a transport frame that Maka does not validate, so the case is live there. It is also wrong in principle: this record is the model's own arguments, and the action is one of them. Collapsing an unrecognised name is computerUseApprovalSummary's job, where knownAction decides what a person is asked to allow; a test now pins that the two functions disagree on purpose. The action is bounded and redacted like every other model-supplied string.

The declared-and-unconsumed set is dropped. launchApp, CuLaunchedApp, truncated, actions, subrole, placeholder, focused, menu, query, appAlias, obscuringRects, targetWindowId, targetStacking, messageIsAppTextFree, and the scroll_element and window_action members of CuSemanticAction — zero producers and zero consumers, confirmed by grep across packages and apps. The reason is the parity guard's own header: window_action sat in the strict union and not on the wire, was unreachable from the day it shipped, and nothing failed, because a declaration with no producer has nothing to check it. This branch was putting window_action back into that state with scroll_element beside it. computer-use-types.ts is now byte-identical to main; the additions come with the executor that produces them, where a guard can assert they arrive. The PR body has been rewritten so it no longer claims them.

What is left is what has a consumer: COMPUTER_USE_SEMANTIC_ACTIONS with the two-way parity check, and computerUseModelCallArgs wired into the Pi backend.

Negative controls, run rather than asserted. Reverting the geometry branch turned the coordinate tests red on coordinate: '<point>' against [412, 88]; restoring the APPROVAL_ACTIONS gate on action turned the third red on unknown against element_sequence, and the Pi backend's own projection test with it. Both back to green after restoring.

One thing I did not change, because it is outside this branch: tool-runtime.ts:776 still writes computerUseApprovalSummary(permissionArgs) into both the tool_start event and the persisted tool_call message on the AI SDK path. That is the same defect this PR fixes, on the main backend rather than the Pi one. It needs its own change, since the same value also feeds describeToolIntent, the durable args hash and the approval record.

Test delta against a clean rebuild of main on this machine: none in packages/core or the Computer Use tests. The rest of the run is shell, filesystem and host-process timing under load 40 on this box; each one passes when its file is run alone.

@hqhq1025hqhq1025 changed the title feat(cu): declare the capabilities an executor can have, and the evidence it returnsfix(cu): let a model read back the Computer Use call it actually madeAug 3, 2026
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 0fcfb46 to 03fe3d9CompareAugust 3, 2026 11:11
@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Blocking finding addressed: the fix is moved to the path the shipping app runs.

It was wired into PiAgentBackend, which the desktop app never instantiates.
boot.ts registers only ai-sdk and fake; the pi backend is reachable only from
harbor-cell.ts behind MAKA_PI_*, driving an external CLI, and it does not
construct a ToolRuntime at all. tool-runtime.ts had a zero-byte diff, so on the
shipping path nothing changed. Measured on the real ToolRuntime, a model that
sent action press_key with app, window_id 41, observation_id obs-7, element_id
e12 and text cmd+s read back action, approvalClass, rememberForTurnAllowed,
app, windowId 41, observationId — a key the tool rejects, two fields it never
sent, no element, and a press_key with no key. That is now the one line the PR
changes in production.

The new assertion is on the tool_start event, which is what model-history.ts
replays, rather than on the projection function. Reverted to
computerUseApprovalSummary and rebuilt, it fails with exactly the diff above;
restored, it passes. A test calling computerUseModelCallArgs directly stays
green either way, which is how this survived being fixed once already.

On the coupling with #1888 and the privacy boundary, since the description
asked that it not be resolved silently.

computer-use-privacy-boundary.test.ts is not new to #1888; it has asserted this
line since #1581, and #1888 added an elementId case to it. Both PRs are
therefore changing the same invariant, and this one changes it in place rather
than around it, so what crosses is visible in the diff and is spelled out in
the file, the commit message and the PR body.

Two things now cross. element_id, which is an index the model chose into an
observation it had already read, admitted only when it is a stable identifier
so an accessibility label arriving under that key is dropped — this is exactly
what #1888 wanted and it is subsumed here, in the tool's own casing.
coordinate, start_coordinate and region, which are geometry the model wrote
into the call itself, integers only so a mistyped value still degrades to a
shape.

The argument for carrying coordinates is that they are the model's own output,
already present in the assistant turn that produced them, and that withholding
them leaves a model that clicked a point and missed unable to see that it had
already tried that point. The argument against, that for set_value and type
these are what the user was having typed on their behalf, does not apply to
coordinates: it applies to text and value, and those still do not cross. They
keep their key and lose their value to , because the key has to stay or
the model reads its own history as a call it never made. text for press_key,
key, hold_key and secondary_action is a name from a closed set the executor
publishes, so it is carried; that is one argument name meaning six things, and
only two of the six come from outside the model.

element_identity is added to the host-only set. The Computer Use tool's own
permissionArgs resolves the model's element_id against the live observation and
attaches its identity, and permissionArgs is what this projection now runs on,
so without that the model would read back a key it cannot send whose value came
off the accessibility tree.

For #1888 specifically: when the two are merged, computer-action-label.ts
should read window_id and element_id rather than windowId and elementId, since
the persisted args are now in the tool's dialect. Nothing on main reads the
persisted Computer Use args in the approval-summary dialect — the permission
prompt reads permissionArgs and the approval scope key is computed from the raw
call — so that renderer is the only reader affected.

Non-blocking finding: the title and body are rewritten. 0fcfb46 removed the
capability declarations, so the branch no longer does what it was named for;
what remains is the model-facing projection and the action-list parity guard,
and the description now says so and says what was dropped and why.

Rebased onto current main, cleanly. @maka/core and @maka/runtime suites pass,
npx biome check is clean on the touched files, and node scripts/check-console.mjs
passes.

…ence it returns
Types only. Nothing selects them, nothing behaves differently, and every
existing test passes unchanged — this is the vocabulary a second executor
backend needs before it can be written against the same interface.
What it adds:
- `launchApp` on `CuDispatchBackend`. Starting an application is something an
executor can do and the host could not ask for, so a model that needed a
closed app had no move but to tell the user to open it.
- `truncated` on `CuObservation` and `actions` on `CuObservedElement`. A tree
that was cut short and an element that offers more than a press are both
facts the executor knows and the host had no field to carry, so they were
read, validated and then dropped.
- `COMPUTER_USE_SEMANTIC_ACTIONS`, so the set of element actions has one
written-down list rather than being spelled again at each place that
switches on it.
Optional throughout: a backend that provides none of it satisfies the
interface exactly as before.
… accepts
The previous version of this branch said nothing behaved differently. Four
things did.
`COMPUTER_USE_SEMANTIC_ACTIONS` declared `launch_app`, `scroll_element`,
`element_sequence` and `window_action`. None of those is in the tool's wire
enum on main; they exist only on the branch that adds the executor. The
constant feeds `APPROVAL_ACTIONS`, so `computerUseApprovalSummary` flipped
`knownAction` to true for four names the SDK rejects before the tool ever
runs, and `rememberForTurnAllowed` could evaluate true for two of them. A
person would have read an approval for an action that did not happen. The
constant is now scoped to what this schema accepts. Declaring an executor's
vocabulary is worth doing on the branch that also ships the executor.
`dispatch_refused` had no producer, and adding it to `COMPUTER_USE_ERROR_CODES`
widened `isComputerUseErrorCode` — the filter that decides whether a raw string
from the driver process is passed through or replaced with `capture_failed`.
That is a live trust boundary, not a declaration. Removed until something
produces it. The unproduced fields on the observation and outcome types are
left as they are: they have no runtime effect, and declaring the shape the
executor returns is what this branch is for.
The drift guard the constant's rationale rests on did not exist. The comment
promised a file that was never written, and the guard on main only walks from
the schemas to the catalog, which is the direction that catches a missing name
— not the direction this branch broke. `computer-use-schema-parity.test.ts`
now compares the two in both directions, with a negative control that runs the
real comparator over a fixed catalog so it keeps proving something on the day
the real one is correct.
`ComputerUseSettings` declared `enabled: false` and was registered nowhere:
not in `AppSettings`, not in `UpdateAppSettingsInput`, not in the defaults,
merge or normalize passes that every peer settings module goes through. In the
shipped app Computer Use tools are assembled with no enable check at all, so
the type told a reader the capability is off by default while it is always on.
Dropped. Wiring a real switch is a change with a UI and a migration, and it
does not belong under a types commit.
The model-call projection could not be used. Only `COMPUTER_USE_SEMANTIC_ACTIONS`
was exported from the package entry, so `computerUseModelCallArgs` — the
headline fix — was unreachable from `@maka/runtime`, and the defective
projection it exists to replace stayed in place. It is exported now and wired
into `projectPiToolArgs`, which is what writes the persisted `tool_call`
message and the `tool_start` event: the record the model reads back as its own
call. It had been reading the host's approval summary, which renames
`window_id` to `windowId`, adds two fields the model never sent, and drops
every argument it does not need.
And the projection withheld the wrong half of `text`. It keyed on argument
name, but `text` is six arguments under one name: the key for `press_key`,
`key` and `hold_key`, the element action name for `secondary_action`, a
substring of the window for `select_text`, and typed content for `type`.
Excluding all six is right for two of them. For the rest the model was reading
back `press_key ... text: <text>` and could not see which key it had pressed —
the exact failure the comment cited as motivation. It now keys on the action.
An action the tool does not accept has no plain set, so it falls through to
shapes rather than to whichever set was used last.
…ations nothing produces
The projection withheld coordinates on the grounds that they are
screen-derived. They are not: a coordinate is the model's own output,
four digits it chose and sent. Reduced to `<point>`, a model that clicked
[412, 88] and missed could not tell whether it had already tried that
point — the repeated-call shape this projection exists to make visible,
reintroduced by the projection itself. `coordinate`, `start_coordinate`
and `region` now come back whole when they are integers, and still
degrade to a shape when they are not.
An action the schema does not carry was recorded as `unknown`, so a model
whose call was rejected for the name it sent read its own history as a
call it never made and could not connect the rejection to the cause.
Collapsing an unrecognised name is `computerUseApprovalSummary`'s job,
where `knownAction` decides what a person is asked to allow; this record
is the model's own arguments, and the action is one of them. It is
bounded and redacted like every other model-supplied string.
The rest of the branch declared fourteen fields, one interface, one
backend method and two union members that nothing in the repository sets
and nothing reads. The parity guard added here covers the action-name
catalog and nothing else, so those declarations had nothing holding them
honest — which is the same condition the guard's own header describes:
`window_action` sat in the strict union and not on the wire and was
unreachable from the day it shipped, and nothing failed. This branch was
putting `window_action` back in that state, along with `scroll_element`.
They come with the executor that produces them, where a guard can check
that they arrive.
What is left is what has a consumer: `COMPUTER_USE_SEMANTIC_ACTIONS` with
the two-way parity check that keeps it and the wire enum in step, and
`computerUseModelCallArgs` wired into the Pi backend's tool_call record.
The projection was wired into PiAgentBackend, which the desktop app never
instantiates: boot.ts registers only ai-sdk and fake, and the pi backend
is reachable only from harbor-cell.ts behind MAKA_PI_*, driving an
external CLI, and it does not build a ToolRuntime at all. tool-runtime.ts
had a zero-byte diff, so on the shipping path nothing changed.
Measured on the real ToolRuntime before this commit, for a model that
sent {action:'press_key', app:'TextEdit', window_id:41,
observation_id:'obs-7', element_id:'e12', text:'cmd+s'}, the persisted
tool_start args were {action, approvalClass, rememberForTurnAllowed, app,
windowId:41, observationId} — a key the tool rejects, two host fields the
model never sent, no element, and a press_key with no key. model-history
replays event.content.args, so that is what the model read back of its
own call, and it went on calling it that way.
The line moves; the projection does not. The permission prompt still
reads permissionArgs and the approval scope key is still computed from
the raw call, so only what is written down changes.
Two things now cross the privacy boundary that did not, and
computer-use-privacy-boundary.test.ts says so where it asserts them.
element_id: an index the model chose into an observation it had already
read. Admitted only when it is a stable identifier, so an accessibility
label arriving under that key is dropped rather than persisted. Without
it every element action in a turn reads the same.
coordinate, start_coordinate, region: geometry the model wrote into the
call itself, integers only, so a mistyped value still degrades to a
shape. A coordinate is not read off the screen; withholding it left a
model that clicked a point and missed unable to see that it had already
tried that point.
What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed — those keep their key
and lose their value to <text>. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so
it is carried; that is one argument name meaning six things, and only two
of the six come from outside the model.
element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the
model's element_id against the live observation and attaches it, and
permissionArgs is what this projection now runs on, so without that the
model would read back a key it cannot send whose value came off the
accessibility tree.
The new assertion is on the tool_start event, not on the projection
function. A test that calls computerUseModelCallArgs directly stays green
while the runtime writes the approval summary, which is how this survived
being fixed once already.
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 03fe3d9 to 71df4c1CompareAugust 3, 2026 11:48
@hqhq1025
hqhq1025 merged commit 80eb37d into apache:mainAug 3, 2026
10 checks passed
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit that referenced this pull request Aug 3, 2026
…1888)
* feat(ui): give driving the computer its own row, and say what it did
Computer Use's display name is a noun — "Maka Computer" — so a turn that
observed a window, clicked a button and observed again rendered three
identical rows. `computerActionLabel` derives a row label from the call's
own arguments instead: 观察「计算器」窗口, 点击元素 e7.
The label is derived, never declared. The model is not given an `intent`
field to write; every word it produces is owned by the runtime, and a free
text field would be one more place it can be wrong.
What the renderer can read is `computerUseApprovalSummary(...)`, which
`ToolRuntime.executeTool` substitutes for the wire call before anything is
persisted. `element_id` had to be carried on that projection for a row to
name its target at all, and this adds it — but a shape filter is not a
privacy boundary. `stableIdentifier` admits `[A-Za-z0-9._:-]{1,256}`, which
is also the shape of an API key, and arguments are not validated before the
projection runs, so a model that put a token under `element_id` wrote it
verbatim into the persisted `tool_call`, onto `tool_start` and into the row
— while the same string under `observation_id` came out `[redacted]`. It
now takes the same `redactSecrets` pass `observationId` and `app` do.
Also puts `'computer'` on the wire as a tool activity kind and has the
Session decoder read the one list rather than a second hand-written copy of
it, which had already fallen behind and rejected the new kind outright.
Note for clients behind the runtime-host Session protocol: the `tool_start`
frame carries no `args`, so such a client gets the kind but cannot derive
the label. The desktop receives full SessionEvents over IPC and is
unaffected.
* fix(ui): read the Computer Use row's arguments in the dialect they are now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. #1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after #1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hqhq1025
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
fix(cu): let a model read back the Computer Use call it actually made by hqhq1025 · Pull Request #1891 · apache/maka · GitHub
Skip to content

fix(cu): let a model read back the Computer Use call it actually made - #1891

Merged
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types
Aug 3, 2026
Merged

fix(cu): let a model read back the Computer Use call it actually made#1891
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types

Conversation

@hqhq1025

@hqhq1025hqhq1025 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The record a model reads back of its own Computer Use call was the host's
approval projection, not the call. model-history.ts replays
event.content.args, and those args came from computerUseApprovalSummary,
which exists to decide and display a permission: it renames window_id to
windowId, adds approvalClass and rememberForTurnAllowed, and drops every
argument it does not need.

Measured on the real ToolRuntime, a model that sent

{"action":"press_key","app":"TextEdit","window_id":41,
"observation_id":"obs-7","element_id":"e12","text":"cmd+s"}

read back

{"action":"press_key","approvalClass":"keyboard_mutation",
"rememberForTurnAllowed":true,"app":"TextEdit","windowId":41,
"observationId":"obs-7"}

— a key the tool rejects, two fields it never sent, no element, and a
press_key with no key. A model imitates the shape it is shown, so it went on
calling it that way; a real desktop run failed six of eleven calls on shapes
copied from its own history.

This adds computerUseModelCallArgs: the same privacy rule, in the names the
tool accepts, and wires it at tool-runtime.ts, which is the line every
shipping backend goes through. The permission prompt still reads
permissionArgs and the approval scope key is still computed from the raw
call, so only what is written down changes.

What now crosses the privacy boundary

computer-use-privacy-boundary.test.ts asserted this line before this PR, and
it is updated in place rather than around, so the change is visible in the
diff. Two things cross that did not:

  • element_id — an index the model chose into an observation it had already
    read (e12). Admitted only when it is a stable identifier, so an
    accessibility label arriving under that key is dropped rather than persisted.
    Without it, ten element actions in a turn are indistinguishable.
  • coordinate, start_coordinate, region — geometry the model wrote into
    the call itself. Integers only, so a mistyped value still degrades to a
    shape. A coordinate is not read off the screen; withholding it leaves a model
    that clicked a point and missed unable to see that it had already tried that
    point.

What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed. Those keep their key and
lose their value to <text> — the key has to stay, or the model reads its own
history as a call it never made. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so it is
carried: one argument name meaning six things, and only two of the six come
from outside the model.

element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the model's
element_id against the live observation and attaches it, and permissionArgs
is what this projection runs on.

Also here

COMPUTER_USE_SEMANTIC_ACTIONS replaces a hand-written action list that had
already drifted: window_action was added to the strict union and not to the
list, so every window move, resize and minimise would have summarised as
unknown — in the approval a person reads, and in the record the model reads
back. computer-use-schema-parity.test.ts now compares the two in both
directions.

Relationship to #1888

#1888 reached the same conclusion from the renderer's side and carried
elementId in the approval summary so its labels could tell two element rows
apart. That intent is subsumed here: element_id is carried, in the tool's own
casing. When the two are merged, #1888's computer-action-label.ts should read
window_id and element_id rather than windowId and elementId, and the
elementId case it added to computer-use-privacy-boundary.test.ts is covered
by the projection assertion in this branch.

Not here any more

The executor capability declarations this PR was originally titled for were
removed in 0fcfb46: nothing produced them, so they were a type surface with
no writer. The title and this description are rewritten to match what the
branch actually does.

Verification

The new assertion is on the tool_start event, not on the projection function.
Reverted to computerUseApprovalSummary at tool-runtime.ts and rebuilt, it
fails with exactly the diff quoted above; restored, it passes. A test that
calls computerUseModelCallArgs directly stays green either way, which is how
this survived being fixed once already in PiAgentBackend — a backend the
desktop app never instantiates.

@maka/core and @maka/runtime suites pass; npx biome check clean on the
touched files; node scripts/check-console.mjs passes. Rebased onto current
main, cleanly.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all six findings. The claim in the PR body that nothing behaves differently was wrong, and finding 1 was the reason.

Finding 1, resolution: I scoped the constant to what main's schema accepts rather than stacking the PR on the executor branch. Verified first — main's wire enum at computer-use-tools.ts is list_apps, observe, click_element, set_value, select_text, secondary_action, press_key plus CU_ACTION_TYPES, and none of launch_app, scroll_element, element_sequence or window_action is in it. The four names went into APPROVAL_ACTIONS through COMPUTER_USE_SEMANTIC_ACTIONS, so computerUseApprovalSummary flipped knownAction to true for calls the SDK rejects before the tool runs, and rememberForTurnAllowed could evaluate true for the two that also joined SEMANTIC_ACTIONS. I chose scoping over stacking for two reasons. The constant's stated purpose is to be held against the tool schema, so declaring names the schema does not have is the same drift it exists to prevent, only in the direction nothing was checking. And stacking would make the PR unmergeable until the executor lands, which delays the parts that are correct today. The executor's vocabulary belongs on the branch that also ships the executor.

Finding 2: the promised file did not exist, but a guard did — computer-use-schema-parity.test.ts, added in #1870 after this branch was cut. It only walks from the schemas to the catalog, which catches a missing name and not an extra one, so the widening in finding 1 would have passed it. I added the reverse comparison to that file rather than creating the second one the comment named, and corrected the comment to point at the real file. Negative control: reintroducing the four names turns the new check red and it names all four. There is also an in-file control that runs the same comparator over a fixed catalog, so it keeps proving the comparator can return something on the day the real catalog is correct.

Finding 3: confirmed. Only COMPUTER_USE_SEMANTIC_ACTIONS was exported, so computerUseModelCallArgs was unreachable from @maka/runtime and the whole headline fix was dead code while the harmful widening was the only part that shipped. The five missing symbols are exported now, and the projection is wired into projectPiToolArgs in pi-agent-backend.ts, which writes the persisted tool_call message and the tool_start event. Negative control: reverting that one line turns the pi-agent-backend projection test red with the approval dialect back in the diff.

Finding 4: confirmed and dropped rather than wired. ComputerUseSettings appears in none of AppSettings, UpdateAppSettingsInput, the defaults, the merge or the normalize pass, unlike every peer settings module, and select-backend.ts assembles the CU tools with no enable check anywhere. A type that says enabled: false while the shipped app always has it on is worse than no type. Wiring a real switch needs a UI and a migration and does not belong under a types commit.

Finding 5: confirmed, and it is exactly as described. text is six arguments under one name — the key for press_key, key and hold_key, the element action name for secondary_action, a screen substring for select_text, typed content for type. MODEL_CALL_PLAIN_VALUES is now a map from action to the argument names that are plain for that action. An action the tool does not accept has no entry, so it falls through to shapes rather than to whichever set was used last. Negative control: reverting to the name-keyed set turns two tests red, showing press_key coming back as text: .

Finding 6, partially rejected with evidence. dispatch_refused is gone. It is not a declaration — COMPUTER_USE_ERROR_CODES is a runtime value feeding isComputerUseErrorCode, which decides at cua-driver-result.ts:88 whether a raw string from the driver process is trusted or replaced with capture_failed. Widening that with no producer is a live change. The other five named items are different in kind. messageIsAppTextFree, truncated, actions, subrole and placeholder are optional fields on TypeScript interfaces with no runtime effect, and they are not distinguishable from about ten other unproduced declarations in the same diff that the review accepted — focused, appAlias, obscuringRects, menu, query, CuLaunchedApp, scroll_element, window_action, launchApp. Dropping the named five and keeping those would be arbitrary; dropping all of them empties the branch of its purpose. Declaring the shape an executor returns is what this branch is for, so they stay, and the absent-means-withheld polarity on messageIsAppTextFree is kept as suggested.

Also rebased onto current main, which the branch was one commit behind — the missing commit was #1870, the parity test finding 2 turns out to depend on.

Test delta, measured against a baseline taken on clean origin/main on the same machine. This machine's Node has no node:sqlite, so @maka/runtime carries 64 pre-existing failing test names there. After the change: 2722 tests, zero new failures and zero fixed against that set. @maka/core goes 703/703 to 710/710, the seven new ones being the model-call projection tests. biome check clean on the six touched files, check-console passes.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Second-round review addressed. Two findings fixed, one dropped after checking, one scope decision.

Coordinates are back. The doc said typed text, written values and coordinates are screen-derived and stay out, and that is false for the third one: a coordinate is the model's own output, four digits it chose and sent. Reduced to <point>, a model that clicked [412, 88] and missed could not tell whether it had already tried that point, which is the repeated-call shape this projection exists to make visible. coordinate, start_coordinate and region now come back whole when they are integers and still degrade to a shape when they are not, so a malformed argument does not become a leak channel.

The out-of-enum action is reported as sent. On the AI SDK path the question is moot — a schema-rejected call never produces a tool-call part, so settleToolCall is never reached and no tool_start is emitted. But this projection's only consumer is the Pi backend, where tool_start arrives as a transport frame that Maka does not validate, so the case is live there. It is also wrong in principle: this record is the model's own arguments, and the action is one of them. Collapsing an unrecognised name is computerUseApprovalSummary's job, where knownAction decides what a person is asked to allow; a test now pins that the two functions disagree on purpose. The action is bounded and redacted like every other model-supplied string.

The declared-and-unconsumed set is dropped. launchApp, CuLaunchedApp, truncated, actions, subrole, placeholder, focused, menu, query, appAlias, obscuringRects, targetWindowId, targetStacking, messageIsAppTextFree, and the scroll_element and window_action members of CuSemanticAction — zero producers and zero consumers, confirmed by grep across packages and apps. The reason is the parity guard's own header: window_action sat in the strict union and not on the wire, was unreachable from the day it shipped, and nothing failed, because a declaration with no producer has nothing to check it. This branch was putting window_action back into that state with scroll_element beside it. computer-use-types.ts is now byte-identical to main; the additions come with the executor that produces them, where a guard can assert they arrive. The PR body has been rewritten so it no longer claims them.

What is left is what has a consumer: COMPUTER_USE_SEMANTIC_ACTIONS with the two-way parity check, and computerUseModelCallArgs wired into the Pi backend.

Negative controls, run rather than asserted. Reverting the geometry branch turned the coordinate tests red on coordinate: '<point>' against [412, 88]; restoring the APPROVAL_ACTIONS gate on action turned the third red on unknown against element_sequence, and the Pi backend's own projection test with it. Both back to green after restoring.

One thing I did not change, because it is outside this branch: tool-runtime.ts:776 still writes computerUseApprovalSummary(permissionArgs) into both the tool_start event and the persisted tool_call message on the AI SDK path. That is the same defect this PR fixes, on the main backend rather than the Pi one. It needs its own change, since the same value also feeds describeToolIntent, the durable args hash and the approval record.

Test delta against a clean rebuild of main on this machine: none in packages/core or the Computer Use tests. The rest of the run is shell, filesystem and host-process timing under load 40 on this box; each one passes when its file is run alone.

@hqhq1025hqhq1025 changed the title feat(cu): declare the capabilities an executor can have, and the evidence it returnsfix(cu): let a model read back the Computer Use call it actually madeAug 3, 2026
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 0fcfb46 to 03fe3d9CompareAugust 3, 2026 11:11
@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Blocking finding addressed: the fix is moved to the path the shipping app runs.

It was wired into PiAgentBackend, which the desktop app never instantiates.
boot.ts registers only ai-sdk and fake; the pi backend is reachable only from
harbor-cell.ts behind MAKA_PI_*, driving an external CLI, and it does not
construct a ToolRuntime at all. tool-runtime.ts had a zero-byte diff, so on the
shipping path nothing changed. Measured on the real ToolRuntime, a model that
sent action press_key with app, window_id 41, observation_id obs-7, element_id
e12 and text cmd+s read back action, approvalClass, rememberForTurnAllowed,
app, windowId 41, observationId — a key the tool rejects, two fields it never
sent, no element, and a press_key with no key. That is now the one line the PR
changes in production.

The new assertion is on the tool_start event, which is what model-history.ts
replays, rather than on the projection function. Reverted to
computerUseApprovalSummary and rebuilt, it fails with exactly the diff above;
restored, it passes. A test calling computerUseModelCallArgs directly stays
green either way, which is how this survived being fixed once already.

On the coupling with #1888 and the privacy boundary, since the description
asked that it not be resolved silently.

computer-use-privacy-boundary.test.ts is not new to #1888; it has asserted this
line since #1581, and #1888 added an elementId case to it. Both PRs are
therefore changing the same invariant, and this one changes it in place rather
than around it, so what crosses is visible in the diff and is spelled out in
the file, the commit message and the PR body.

Two things now cross. element_id, which is an index the model chose into an
observation it had already read, admitted only when it is a stable identifier
so an accessibility label arriving under that key is dropped — this is exactly
what #1888 wanted and it is subsumed here, in the tool's own casing.
coordinate, start_coordinate and region, which are geometry the model wrote
into the call itself, integers only so a mistyped value still degrades to a
shape.

The argument for carrying coordinates is that they are the model's own output,
already present in the assistant turn that produced them, and that withholding
them leaves a model that clicked a point and missed unable to see that it had
already tried that point. The argument against, that for set_value and type
these are what the user was having typed on their behalf, does not apply to
coordinates: it applies to text and value, and those still do not cross. They
keep their key and lose their value to , because the key has to stay or
the model reads its own history as a call it never made. text for press_key,
key, hold_key and secondary_action is a name from a closed set the executor
publishes, so it is carried; that is one argument name meaning six things, and
only two of the six come from outside the model.

element_identity is added to the host-only set. The Computer Use tool's own
permissionArgs resolves the model's element_id against the live observation and
attaches its identity, and permissionArgs is what this projection now runs on,
so without that the model would read back a key it cannot send whose value came
off the accessibility tree.

For #1888 specifically: when the two are merged, computer-action-label.ts
should read window_id and element_id rather than windowId and elementId, since
the persisted args are now in the tool's dialect. Nothing on main reads the
persisted Computer Use args in the approval-summary dialect — the permission
prompt reads permissionArgs and the approval scope key is computed from the raw
call — so that renderer is the only reader affected.

Non-blocking finding: the title and body are rewritten. 0fcfb46 removed the
capability declarations, so the branch no longer does what it was named for;
what remains is the model-facing projection and the action-list parity guard,
and the description now says so and says what was dropped and why.

Rebased onto current main, cleanly. @maka/core and @maka/runtime suites pass,
npx biome check is clean on the touched files, and node scripts/check-console.mjs
passes.

…ence it returns
Types only. Nothing selects them, nothing behaves differently, and every
existing test passes unchanged — this is the vocabulary a second executor
backend needs before it can be written against the same interface.
What it adds:
- `launchApp` on `CuDispatchBackend`. Starting an application is something an
executor can do and the host could not ask for, so a model that needed a
closed app had no move but to tell the user to open it.
- `truncated` on `CuObservation` and `actions` on `CuObservedElement`. A tree
that was cut short and an element that offers more than a press are both
facts the executor knows and the host had no field to carry, so they were
read, validated and then dropped.
- `COMPUTER_USE_SEMANTIC_ACTIONS`, so the set of element actions has one
written-down list rather than being spelled again at each place that
switches on it.
Optional throughout: a backend that provides none of it satisfies the
interface exactly as before.
… accepts
The previous version of this branch said nothing behaved differently. Four
things did.
`COMPUTER_USE_SEMANTIC_ACTIONS` declared `launch_app`, `scroll_element`,
`element_sequence` and `window_action`. None of those is in the tool's wire
enum on main; they exist only on the branch that adds the executor. The
constant feeds `APPROVAL_ACTIONS`, so `computerUseApprovalSummary` flipped
`knownAction` to true for four names the SDK rejects before the tool ever
runs, and `rememberForTurnAllowed` could evaluate true for two of them. A
person would have read an approval for an action that did not happen. The
constant is now scoped to what this schema accepts. Declaring an executor's
vocabulary is worth doing on the branch that also ships the executor.
`dispatch_refused` had no producer, and adding it to `COMPUTER_USE_ERROR_CODES`
widened `isComputerUseErrorCode` — the filter that decides whether a raw string
from the driver process is passed through or replaced with `capture_failed`.
That is a live trust boundary, not a declaration. Removed until something
produces it. The unproduced fields on the observation and outcome types are
left as they are: they have no runtime effect, and declaring the shape the
executor returns is what this branch is for.
The drift guard the constant's rationale rests on did not exist. The comment
promised a file that was never written, and the guard on main only walks from
the schemas to the catalog, which is the direction that catches a missing name
— not the direction this branch broke. `computer-use-schema-parity.test.ts`
now compares the two in both directions, with a negative control that runs the
real comparator over a fixed catalog so it keeps proving something on the day
the real one is correct.
`ComputerUseSettings` declared `enabled: false` and was registered nowhere:
not in `AppSettings`, not in `UpdateAppSettingsInput`, not in the defaults,
merge or normalize passes that every peer settings module goes through. In the
shipped app Computer Use tools are assembled with no enable check at all, so
the type told a reader the capability is off by default while it is always on.
Dropped. Wiring a real switch is a change with a UI and a migration, and it
does not belong under a types commit.
The model-call projection could not be used. Only `COMPUTER_USE_SEMANTIC_ACTIONS`
was exported from the package entry, so `computerUseModelCallArgs` — the
headline fix — was unreachable from `@maka/runtime`, and the defective
projection it exists to replace stayed in place. It is exported now and wired
into `projectPiToolArgs`, which is what writes the persisted `tool_call`
message and the `tool_start` event: the record the model reads back as its own
call. It had been reading the host's approval summary, which renames
`window_id` to `windowId`, adds two fields the model never sent, and drops
every argument it does not need.
And the projection withheld the wrong half of `text`. It keyed on argument
name, but `text` is six arguments under one name: the key for `press_key`,
`key` and `hold_key`, the element action name for `secondary_action`, a
substring of the window for `select_text`, and typed content for `type`.
Excluding all six is right for two of them. For the rest the model was reading
back `press_key ... text: <text>` and could not see which key it had pressed —
the exact failure the comment cited as motivation. It now keys on the action.
An action the tool does not accept has no plain set, so it falls through to
shapes rather than to whichever set was used last.
…ations nothing produces
The projection withheld coordinates on the grounds that they are
screen-derived. They are not: a coordinate is the model's own output,
four digits it chose and sent. Reduced to `<point>`, a model that clicked
[412, 88] and missed could not tell whether it had already tried that
point — the repeated-call shape this projection exists to make visible,
reintroduced by the projection itself. `coordinate`, `start_coordinate`
and `region` now come back whole when they are integers, and still
degrade to a shape when they are not.
An action the schema does not carry was recorded as `unknown`, so a model
whose call was rejected for the name it sent read its own history as a
call it never made and could not connect the rejection to the cause.
Collapsing an unrecognised name is `computerUseApprovalSummary`'s job,
where `knownAction` decides what a person is asked to allow; this record
is the model's own arguments, and the action is one of them. It is
bounded and redacted like every other model-supplied string.
The rest of the branch declared fourteen fields, one interface, one
backend method and two union members that nothing in the repository sets
and nothing reads. The parity guard added here covers the action-name
catalog and nothing else, so those declarations had nothing holding them
honest — which is the same condition the guard's own header describes:
`window_action` sat in the strict union and not on the wire and was
unreachable from the day it shipped, and nothing failed. This branch was
putting `window_action` back in that state, along with `scroll_element`.
They come with the executor that produces them, where a guard can check
that they arrive.
What is left is what has a consumer: `COMPUTER_USE_SEMANTIC_ACTIONS` with
the two-way parity check that keeps it and the wire enum in step, and
`computerUseModelCallArgs` wired into the Pi backend's tool_call record.
The projection was wired into PiAgentBackend, which the desktop app never
instantiates: boot.ts registers only ai-sdk and fake, and the pi backend
is reachable only from harbor-cell.ts behind MAKA_PI_*, driving an
external CLI, and it does not build a ToolRuntime at all. tool-runtime.ts
had a zero-byte diff, so on the shipping path nothing changed.
Measured on the real ToolRuntime before this commit, for a model that
sent {action:'press_key', app:'TextEdit', window_id:41,
observation_id:'obs-7', element_id:'e12', text:'cmd+s'}, the persisted
tool_start args were {action, approvalClass, rememberForTurnAllowed, app,
windowId:41, observationId} — a key the tool rejects, two host fields the
model never sent, no element, and a press_key with no key. model-history
replays event.content.args, so that is what the model read back of its
own call, and it went on calling it that way.
The line moves; the projection does not. The permission prompt still
reads permissionArgs and the approval scope key is still computed from
the raw call, so only what is written down changes.
Two things now cross the privacy boundary that did not, and
computer-use-privacy-boundary.test.ts says so where it asserts them.
element_id: an index the model chose into an observation it had already
read. Admitted only when it is a stable identifier, so an accessibility
label arriving under that key is dropped rather than persisted. Without
it every element action in a turn reads the same.
coordinate, start_coordinate, region: geometry the model wrote into the
call itself, integers only, so a mistyped value still degrades to a
shape. A coordinate is not read off the screen; withholding it left a
model that clicked a point and missed unable to see that it had already
tried that point.
What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed — those keep their key
and lose their value to <text>. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so
it is carried; that is one argument name meaning six things, and only two
of the six come from outside the model.
element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the
model's element_id against the live observation and attaches it, and
permissionArgs is what this projection now runs on, so without that the
model would read back a key it cannot send whose value came off the
accessibility tree.
The new assertion is on the tool_start event, not on the projection
function. A test that calls computerUseModelCallArgs directly stays green
while the runtime writes the approval summary, which is how this survived
being fixed once already.
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 03fe3d9 to 71df4c1CompareAugust 3, 2026 11:48
@hqhq1025
hqhq1025 merged commit 80eb37d into apache:mainAug 3, 2026
10 checks passed
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit that referenced this pull request Aug 3, 2026
…1888)
* feat(ui): give driving the computer its own row, and say what it did
Computer Use's display name is a noun — "Maka Computer" — so a turn that
observed a window, clicked a button and observed again rendered three
identical rows. `computerActionLabel` derives a row label from the call's
own arguments instead: 观察「计算器」窗口, 点击元素 e7.
The label is derived, never declared. The model is not given an `intent`
field to write; every word it produces is owned by the runtime, and a free
text field would be one more place it can be wrong.
What the renderer can read is `computerUseApprovalSummary(...)`, which
`ToolRuntime.executeTool` substitutes for the wire call before anything is
persisted. `element_id` had to be carried on that projection for a row to
name its target at all, and this adds it — but a shape filter is not a
privacy boundary. `stableIdentifier` admits `[A-Za-z0-9._:-]{1,256}`, which
is also the shape of an API key, and arguments are not validated before the
projection runs, so a model that put a token under `element_id` wrote it
verbatim into the persisted `tool_call`, onto `tool_start` and into the row
— while the same string under `observation_id` came out `[redacted]`. It
now takes the same `redactSecrets` pass `observationId` and `app` do.
Also puts `'computer'` on the wire as a tool activity kind and has the
Session decoder read the one list rather than a second hand-written copy of
it, which had already fallen behind and rejected the new kind outright.
Note for clients behind the runtime-host Session protocol: the `tool_start`
frame carries no `args`, so such a client gets the kind but cannot derive
the label. The desktop receives full SessionEvents over IPC and is
unaffected.
* fix(ui): read the Computer Use row's arguments in the dialect they are now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. #1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after #1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hqhq1025
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(cu): let a model read back the Computer Use call it actually made by hqhq1025 · Pull Request #1891 · apache/maka · GitHub
Skip to content

fix(cu): let a model read back the Computer Use call it actually made - #1891

Merged
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types
Aug 3, 2026
Merged

fix(cu): let a model read back the Computer Use call it actually made#1891
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types

Conversation

@hqhq1025

@hqhq1025hqhq1025 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The record a model reads back of its own Computer Use call was the host's
approval projection, not the call. model-history.ts replays
event.content.args, and those args came from computerUseApprovalSummary,
which exists to decide and display a permission: it renames window_id to
windowId, adds approvalClass and rememberForTurnAllowed, and drops every
argument it does not need.

Measured on the real ToolRuntime, a model that sent

{"action":"press_key","app":"TextEdit","window_id":41,
"observation_id":"obs-7","element_id":"e12","text":"cmd+s"}

read back

{"action":"press_key","approvalClass":"keyboard_mutation",
"rememberForTurnAllowed":true,"app":"TextEdit","windowId":41,
"observationId":"obs-7"}

— a key the tool rejects, two fields it never sent, no element, and a
press_key with no key. A model imitates the shape it is shown, so it went on
calling it that way; a real desktop run failed six of eleven calls on shapes
copied from its own history.

This adds computerUseModelCallArgs: the same privacy rule, in the names the
tool accepts, and wires it at tool-runtime.ts, which is the line every
shipping backend goes through. The permission prompt still reads
permissionArgs and the approval scope key is still computed from the raw
call, so only what is written down changes.

What now crosses the privacy boundary

computer-use-privacy-boundary.test.ts asserted this line before this PR, and
it is updated in place rather than around, so the change is visible in the
diff. Two things cross that did not:

  • element_id — an index the model chose into an observation it had already
    read (e12). Admitted only when it is a stable identifier, so an
    accessibility label arriving under that key is dropped rather than persisted.
    Without it, ten element actions in a turn are indistinguishable.
  • coordinate, start_coordinate, region — geometry the model wrote into
    the call itself. Integers only, so a mistyped value still degrades to a
    shape. A coordinate is not read off the screen; withholding it leaves a model
    that clicked a point and missed unable to see that it had already tried that
    point.

What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed. Those keep their key and
lose their value to <text> — the key has to stay, or the model reads its own
history as a call it never made. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so it is
carried: one argument name meaning six things, and only two of the six come
from outside the model.

element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the model's
element_id against the live observation and attaches it, and permissionArgs
is what this projection runs on.

Also here

COMPUTER_USE_SEMANTIC_ACTIONS replaces a hand-written action list that had
already drifted: window_action was added to the strict union and not to the
list, so every window move, resize and minimise would have summarised as
unknown — in the approval a person reads, and in the record the model reads
back. computer-use-schema-parity.test.ts now compares the two in both
directions.

Relationship to #1888

#1888 reached the same conclusion from the renderer's side and carried
elementId in the approval summary so its labels could tell two element rows
apart. That intent is subsumed here: element_id is carried, in the tool's own
casing. When the two are merged, #1888's computer-action-label.ts should read
window_id and element_id rather than windowId and elementId, and the
elementId case it added to computer-use-privacy-boundary.test.ts is covered
by the projection assertion in this branch.

Not here any more

The executor capability declarations this PR was originally titled for were
removed in 0fcfb46: nothing produced them, so they were a type surface with
no writer. The title and this description are rewritten to match what the
branch actually does.

Verification

The new assertion is on the tool_start event, not on the projection function.
Reverted to computerUseApprovalSummary at tool-runtime.ts and rebuilt, it
fails with exactly the diff quoted above; restored, it passes. A test that
calls computerUseModelCallArgs directly stays green either way, which is how
this survived being fixed once already in PiAgentBackend — a backend the
desktop app never instantiates.

@maka/core and @maka/runtime suites pass; npx biome check clean on the
touched files; node scripts/check-console.mjs passes. Rebased onto current
main, cleanly.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all six findings. The claim in the PR body that nothing behaves differently was wrong, and finding 1 was the reason.

Finding 1, resolution: I scoped the constant to what main's schema accepts rather than stacking the PR on the executor branch. Verified first — main's wire enum at computer-use-tools.ts is list_apps, observe, click_element, set_value, select_text, secondary_action, press_key plus CU_ACTION_TYPES, and none of launch_app, scroll_element, element_sequence or window_action is in it. The four names went into APPROVAL_ACTIONS through COMPUTER_USE_SEMANTIC_ACTIONS, so computerUseApprovalSummary flipped knownAction to true for calls the SDK rejects before the tool runs, and rememberForTurnAllowed could evaluate true for the two that also joined SEMANTIC_ACTIONS. I chose scoping over stacking for two reasons. The constant's stated purpose is to be held against the tool schema, so declaring names the schema does not have is the same drift it exists to prevent, only in the direction nothing was checking. And stacking would make the PR unmergeable until the executor lands, which delays the parts that are correct today. The executor's vocabulary belongs on the branch that also ships the executor.

Finding 2: the promised file did not exist, but a guard did — computer-use-schema-parity.test.ts, added in #1870 after this branch was cut. It only walks from the schemas to the catalog, which catches a missing name and not an extra one, so the widening in finding 1 would have passed it. I added the reverse comparison to that file rather than creating the second one the comment named, and corrected the comment to point at the real file. Negative control: reintroducing the four names turns the new check red and it names all four. There is also an in-file control that runs the same comparator over a fixed catalog, so it keeps proving the comparator can return something on the day the real catalog is correct.

Finding 3: confirmed. Only COMPUTER_USE_SEMANTIC_ACTIONS was exported, so computerUseModelCallArgs was unreachable from @maka/runtime and the whole headline fix was dead code while the harmful widening was the only part that shipped. The five missing symbols are exported now, and the projection is wired into projectPiToolArgs in pi-agent-backend.ts, which writes the persisted tool_call message and the tool_start event. Negative control: reverting that one line turns the pi-agent-backend projection test red with the approval dialect back in the diff.

Finding 4: confirmed and dropped rather than wired. ComputerUseSettings appears in none of AppSettings, UpdateAppSettingsInput, the defaults, the merge or the normalize pass, unlike every peer settings module, and select-backend.ts assembles the CU tools with no enable check anywhere. A type that says enabled: false while the shipped app always has it on is worse than no type. Wiring a real switch needs a UI and a migration and does not belong under a types commit.

Finding 5: confirmed, and it is exactly as described. text is six arguments under one name — the key for press_key, key and hold_key, the element action name for secondary_action, a screen substring for select_text, typed content for type. MODEL_CALL_PLAIN_VALUES is now a map from action to the argument names that are plain for that action. An action the tool does not accept has no entry, so it falls through to shapes rather than to whichever set was used last. Negative control: reverting to the name-keyed set turns two tests red, showing press_key coming back as text: .

Finding 6, partially rejected with evidence. dispatch_refused is gone. It is not a declaration — COMPUTER_USE_ERROR_CODES is a runtime value feeding isComputerUseErrorCode, which decides at cua-driver-result.ts:88 whether a raw string from the driver process is trusted or replaced with capture_failed. Widening that with no producer is a live change. The other five named items are different in kind. messageIsAppTextFree, truncated, actions, subrole and placeholder are optional fields on TypeScript interfaces with no runtime effect, and they are not distinguishable from about ten other unproduced declarations in the same diff that the review accepted — focused, appAlias, obscuringRects, menu, query, CuLaunchedApp, scroll_element, window_action, launchApp. Dropping the named five and keeping those would be arbitrary; dropping all of them empties the branch of its purpose. Declaring the shape an executor returns is what this branch is for, so they stay, and the absent-means-withheld polarity on messageIsAppTextFree is kept as suggested.

Also rebased onto current main, which the branch was one commit behind — the missing commit was #1870, the parity test finding 2 turns out to depend on.

Test delta, measured against a baseline taken on clean origin/main on the same machine. This machine's Node has no node:sqlite, so @maka/runtime carries 64 pre-existing failing test names there. After the change: 2722 tests, zero new failures and zero fixed against that set. @maka/core goes 703/703 to 710/710, the seven new ones being the model-call projection tests. biome check clean on the six touched files, check-console passes.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Second-round review addressed. Two findings fixed, one dropped after checking, one scope decision.

Coordinates are back. The doc said typed text, written values and coordinates are screen-derived and stay out, and that is false for the third one: a coordinate is the model's own output, four digits it chose and sent. Reduced to <point>, a model that clicked [412, 88] and missed could not tell whether it had already tried that point, which is the repeated-call shape this projection exists to make visible. coordinate, start_coordinate and region now come back whole when they are integers and still degrade to a shape when they are not, so a malformed argument does not become a leak channel.

The out-of-enum action is reported as sent. On the AI SDK path the question is moot — a schema-rejected call never produces a tool-call part, so settleToolCall is never reached and no tool_start is emitted. But this projection's only consumer is the Pi backend, where tool_start arrives as a transport frame that Maka does not validate, so the case is live there. It is also wrong in principle: this record is the model's own arguments, and the action is one of them. Collapsing an unrecognised name is computerUseApprovalSummary's job, where knownAction decides what a person is asked to allow; a test now pins that the two functions disagree on purpose. The action is bounded and redacted like every other model-supplied string.

The declared-and-unconsumed set is dropped. launchApp, CuLaunchedApp, truncated, actions, subrole, placeholder, focused, menu, query, appAlias, obscuringRects, targetWindowId, targetStacking, messageIsAppTextFree, and the scroll_element and window_action members of CuSemanticAction — zero producers and zero consumers, confirmed by grep across packages and apps. The reason is the parity guard's own header: window_action sat in the strict union and not on the wire, was unreachable from the day it shipped, and nothing failed, because a declaration with no producer has nothing to check it. This branch was putting window_action back into that state with scroll_element beside it. computer-use-types.ts is now byte-identical to main; the additions come with the executor that produces them, where a guard can assert they arrive. The PR body has been rewritten so it no longer claims them.

What is left is what has a consumer: COMPUTER_USE_SEMANTIC_ACTIONS with the two-way parity check, and computerUseModelCallArgs wired into the Pi backend.

Negative controls, run rather than asserted. Reverting the geometry branch turned the coordinate tests red on coordinate: '<point>' against [412, 88]; restoring the APPROVAL_ACTIONS gate on action turned the third red on unknown against element_sequence, and the Pi backend's own projection test with it. Both back to green after restoring.

One thing I did not change, because it is outside this branch: tool-runtime.ts:776 still writes computerUseApprovalSummary(permissionArgs) into both the tool_start event and the persisted tool_call message on the AI SDK path. That is the same defect this PR fixes, on the main backend rather than the Pi one. It needs its own change, since the same value also feeds describeToolIntent, the durable args hash and the approval record.

Test delta against a clean rebuild of main on this machine: none in packages/core or the Computer Use tests. The rest of the run is shell, filesystem and host-process timing under load 40 on this box; each one passes when its file is run alone.

@hqhq1025hqhq1025 changed the title feat(cu): declare the capabilities an executor can have, and the evidence it returnsfix(cu): let a model read back the Computer Use call it actually madeAug 3, 2026
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 0fcfb46 to 03fe3d9CompareAugust 3, 2026 11:11
@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Blocking finding addressed: the fix is moved to the path the shipping app runs.

It was wired into PiAgentBackend, which the desktop app never instantiates.
boot.ts registers only ai-sdk and fake; the pi backend is reachable only from
harbor-cell.ts behind MAKA_PI_*, driving an external CLI, and it does not
construct a ToolRuntime at all. tool-runtime.ts had a zero-byte diff, so on the
shipping path nothing changed. Measured on the real ToolRuntime, a model that
sent action press_key with app, window_id 41, observation_id obs-7, element_id
e12 and text cmd+s read back action, approvalClass, rememberForTurnAllowed,
app, windowId 41, observationId — a key the tool rejects, two fields it never
sent, no element, and a press_key with no key. That is now the one line the PR
changes in production.

The new assertion is on the tool_start event, which is what model-history.ts
replays, rather than on the projection function. Reverted to
computerUseApprovalSummary and rebuilt, it fails with exactly the diff above;
restored, it passes. A test calling computerUseModelCallArgs directly stays
green either way, which is how this survived being fixed once already.

On the coupling with #1888 and the privacy boundary, since the description
asked that it not be resolved silently.

computer-use-privacy-boundary.test.ts is not new to #1888; it has asserted this
line since #1581, and #1888 added an elementId case to it. Both PRs are
therefore changing the same invariant, and this one changes it in place rather
than around it, so what crosses is visible in the diff and is spelled out in
the file, the commit message and the PR body.

Two things now cross. element_id, which is an index the model chose into an
observation it had already read, admitted only when it is a stable identifier
so an accessibility label arriving under that key is dropped — this is exactly
what #1888 wanted and it is subsumed here, in the tool's own casing.
coordinate, start_coordinate and region, which are geometry the model wrote
into the call itself, integers only so a mistyped value still degrades to a
shape.

The argument for carrying coordinates is that they are the model's own output,
already present in the assistant turn that produced them, and that withholding
them leaves a model that clicked a point and missed unable to see that it had
already tried that point. The argument against, that for set_value and type
these are what the user was having typed on their behalf, does not apply to
coordinates: it applies to text and value, and those still do not cross. They
keep their key and lose their value to , because the key has to stay or
the model reads its own history as a call it never made. text for press_key,
key, hold_key and secondary_action is a name from a closed set the executor
publishes, so it is carried; that is one argument name meaning six things, and
only two of the six come from outside the model.

element_identity is added to the host-only set. The Computer Use tool's own
permissionArgs resolves the model's element_id against the live observation and
attaches its identity, and permissionArgs is what this projection now runs on,
so without that the model would read back a key it cannot send whose value came
off the accessibility tree.

For #1888 specifically: when the two are merged, computer-action-label.ts
should read window_id and element_id rather than windowId and elementId, since
the persisted args are now in the tool's dialect. Nothing on main reads the
persisted Computer Use args in the approval-summary dialect — the permission
prompt reads permissionArgs and the approval scope key is computed from the raw
call — so that renderer is the only reader affected.

Non-blocking finding: the title and body are rewritten. 0fcfb46 removed the
capability declarations, so the branch no longer does what it was named for;
what remains is the model-facing projection and the action-list parity guard,
and the description now says so and says what was dropped and why.

Rebased onto current main, cleanly. @maka/core and @maka/runtime suites pass,
npx biome check is clean on the touched files, and node scripts/check-console.mjs
passes.

…ence it returns
Types only. Nothing selects them, nothing behaves differently, and every
existing test passes unchanged — this is the vocabulary a second executor
backend needs before it can be written against the same interface.
What it adds:
- `launchApp` on `CuDispatchBackend`. Starting an application is something an
executor can do and the host could not ask for, so a model that needed a
closed app had no move but to tell the user to open it.
- `truncated` on `CuObservation` and `actions` on `CuObservedElement`. A tree
that was cut short and an element that offers more than a press are both
facts the executor knows and the host had no field to carry, so they were
read, validated and then dropped.
- `COMPUTER_USE_SEMANTIC_ACTIONS`, so the set of element actions has one
written-down list rather than being spelled again at each place that
switches on it.
Optional throughout: a backend that provides none of it satisfies the
interface exactly as before.
… accepts
The previous version of this branch said nothing behaved differently. Four
things did.
`COMPUTER_USE_SEMANTIC_ACTIONS` declared `launch_app`, `scroll_element`,
`element_sequence` and `window_action`. None of those is in the tool's wire
enum on main; they exist only on the branch that adds the executor. The
constant feeds `APPROVAL_ACTIONS`, so `computerUseApprovalSummary` flipped
`knownAction` to true for four names the SDK rejects before the tool ever
runs, and `rememberForTurnAllowed` could evaluate true for two of them. A
person would have read an approval for an action that did not happen. The
constant is now scoped to what this schema accepts. Declaring an executor's
vocabulary is worth doing on the branch that also ships the executor.
`dispatch_refused` had no producer, and adding it to `COMPUTER_USE_ERROR_CODES`
widened `isComputerUseErrorCode` — the filter that decides whether a raw string
from the driver process is passed through or replaced with `capture_failed`.
That is a live trust boundary, not a declaration. Removed until something
produces it. The unproduced fields on the observation and outcome types are
left as they are: they have no runtime effect, and declaring the shape the
executor returns is what this branch is for.
The drift guard the constant's rationale rests on did not exist. The comment
promised a file that was never written, and the guard on main only walks from
the schemas to the catalog, which is the direction that catches a missing name
— not the direction this branch broke. `computer-use-schema-parity.test.ts`
now compares the two in both directions, with a negative control that runs the
real comparator over a fixed catalog so it keeps proving something on the day
the real one is correct.
`ComputerUseSettings` declared `enabled: false` and was registered nowhere:
not in `AppSettings`, not in `UpdateAppSettingsInput`, not in the defaults,
merge or normalize passes that every peer settings module goes through. In the
shipped app Computer Use tools are assembled with no enable check at all, so
the type told a reader the capability is off by default while it is always on.
Dropped. Wiring a real switch is a change with a UI and a migration, and it
does not belong under a types commit.
The model-call projection could not be used. Only `COMPUTER_USE_SEMANTIC_ACTIONS`
was exported from the package entry, so `computerUseModelCallArgs` — the
headline fix — was unreachable from `@maka/runtime`, and the defective
projection it exists to replace stayed in place. It is exported now and wired
into `projectPiToolArgs`, which is what writes the persisted `tool_call`
message and the `tool_start` event: the record the model reads back as its own
call. It had been reading the host's approval summary, which renames
`window_id` to `windowId`, adds two fields the model never sent, and drops
every argument it does not need.
And the projection withheld the wrong half of `text`. It keyed on argument
name, but `text` is six arguments under one name: the key for `press_key`,
`key` and `hold_key`, the element action name for `secondary_action`, a
substring of the window for `select_text`, and typed content for `type`.
Excluding all six is right for two of them. For the rest the model was reading
back `press_key ... text: <text>` and could not see which key it had pressed —
the exact failure the comment cited as motivation. It now keys on the action.
An action the tool does not accept has no plain set, so it falls through to
shapes rather than to whichever set was used last.
…ations nothing produces
The projection withheld coordinates on the grounds that they are
screen-derived. They are not: a coordinate is the model's own output,
four digits it chose and sent. Reduced to `<point>`, a model that clicked
[412, 88] and missed could not tell whether it had already tried that
point — the repeated-call shape this projection exists to make visible,
reintroduced by the projection itself. `coordinate`, `start_coordinate`
and `region` now come back whole when they are integers, and still
degrade to a shape when they are not.
An action the schema does not carry was recorded as `unknown`, so a model
whose call was rejected for the name it sent read its own history as a
call it never made and could not connect the rejection to the cause.
Collapsing an unrecognised name is `computerUseApprovalSummary`'s job,
where `knownAction` decides what a person is asked to allow; this record
is the model's own arguments, and the action is one of them. It is
bounded and redacted like every other model-supplied string.
The rest of the branch declared fourteen fields, one interface, one
backend method and two union members that nothing in the repository sets
and nothing reads. The parity guard added here covers the action-name
catalog and nothing else, so those declarations had nothing holding them
honest — which is the same condition the guard's own header describes:
`window_action` sat in the strict union and not on the wire and was
unreachable from the day it shipped, and nothing failed. This branch was
putting `window_action` back in that state, along with `scroll_element`.
They come with the executor that produces them, where a guard can check
that they arrive.
What is left is what has a consumer: `COMPUTER_USE_SEMANTIC_ACTIONS` with
the two-way parity check that keeps it and the wire enum in step, and
`computerUseModelCallArgs` wired into the Pi backend's tool_call record.
The projection was wired into PiAgentBackend, which the desktop app never
instantiates: boot.ts registers only ai-sdk and fake, and the pi backend
is reachable only from harbor-cell.ts behind MAKA_PI_*, driving an
external CLI, and it does not build a ToolRuntime at all. tool-runtime.ts
had a zero-byte diff, so on the shipping path nothing changed.
Measured on the real ToolRuntime before this commit, for a model that
sent {action:'press_key', app:'TextEdit', window_id:41,
observation_id:'obs-7', element_id:'e12', text:'cmd+s'}, the persisted
tool_start args were {action, approvalClass, rememberForTurnAllowed, app,
windowId:41, observationId} — a key the tool rejects, two host fields the
model never sent, no element, and a press_key with no key. model-history
replays event.content.args, so that is what the model read back of its
own call, and it went on calling it that way.
The line moves; the projection does not. The permission prompt still
reads permissionArgs and the approval scope key is still computed from
the raw call, so only what is written down changes.
Two things now cross the privacy boundary that did not, and
computer-use-privacy-boundary.test.ts says so where it asserts them.
element_id: an index the model chose into an observation it had already
read. Admitted only when it is a stable identifier, so an accessibility
label arriving under that key is dropped rather than persisted. Without
it every element action in a turn reads the same.
coordinate, start_coordinate, region: geometry the model wrote into the
call itself, integers only, so a mistyped value still degrades to a
shape. A coordinate is not read off the screen; withholding it left a
model that clicked a point and missed unable to see that it had already
tried that point.
What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed — those keep their key
and lose their value to <text>. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so
it is carried; that is one argument name meaning six things, and only two
of the six come from outside the model.
element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the
model's element_id against the live observation and attaches it, and
permissionArgs is what this projection now runs on, so without that the
model would read back a key it cannot send whose value came off the
accessibility tree.
The new assertion is on the tool_start event, not on the projection
function. A test that calls computerUseModelCallArgs directly stays green
while the runtime writes the approval summary, which is how this survived
being fixed once already.
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 03fe3d9 to 71df4c1CompareAugust 3, 2026 11:48
@hqhq1025
hqhq1025 merged commit 80eb37d into apache:mainAug 3, 2026
10 checks passed
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit that referenced this pull request Aug 3, 2026
…1888)
* feat(ui): give driving the computer its own row, and say what it did
Computer Use's display name is a noun — "Maka Computer" — so a turn that
observed a window, clicked a button and observed again rendered three
identical rows. `computerActionLabel` derives a row label from the call's
own arguments instead: 观察「计算器」窗口, 点击元素 e7.
The label is derived, never declared. The model is not given an `intent`
field to write; every word it produces is owned by the runtime, and a free
text field would be one more place it can be wrong.
What the renderer can read is `computerUseApprovalSummary(...)`, which
`ToolRuntime.executeTool` substitutes for the wire call before anything is
persisted. `element_id` had to be carried on that projection for a row to
name its target at all, and this adds it — but a shape filter is not a
privacy boundary. `stableIdentifier` admits `[A-Za-z0-9._:-]{1,256}`, which
is also the shape of an API key, and arguments are not validated before the
projection runs, so a model that put a token under `element_id` wrote it
verbatim into the persisted `tool_call`, onto `tool_start` and into the row
— while the same string under `observation_id` came out `[redacted]`. It
now takes the same `redactSecrets` pass `observationId` and `app` do.
Also puts `'computer'` on the wire as a tool activity kind and has the
Session decoder read the one list rather than a second hand-written copy of
it, which had already fallen behind and rejected the new kind outright.
Note for clients behind the runtime-host Session protocol: the `tool_start`
frame carries no `args`, so such a client gets the kind but cannot derive
the label. The desktop receives full SessionEvents over IPC and is
unaffected.
* fix(ui): read the Computer Use row's arguments in the dialect they are now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. #1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after #1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hqhq1025
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(cu): let a model read back the Computer Use call it actually made by hqhq1025 · Pull Request #1891 · apache/maka · GitHub
Skip to content

fix(cu): let a model read back the Computer Use call it actually made - #1891

Merged
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types
Aug 3, 2026
Merged

fix(cu): let a model read back the Computer Use call it actually made#1891
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types

Conversation

@hqhq1025

@hqhq1025hqhq1025 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The record a model reads back of its own Computer Use call was the host's
approval projection, not the call. model-history.ts replays
event.content.args, and those args came from computerUseApprovalSummary,
which exists to decide and display a permission: it renames window_id to
windowId, adds approvalClass and rememberForTurnAllowed, and drops every
argument it does not need.

Measured on the real ToolRuntime, a model that sent

{"action":"press_key","app":"TextEdit","window_id":41,
"observation_id":"obs-7","element_id":"e12","text":"cmd+s"}

read back

{"action":"press_key","approvalClass":"keyboard_mutation",
"rememberForTurnAllowed":true,"app":"TextEdit","windowId":41,
"observationId":"obs-7"}

— a key the tool rejects, two fields it never sent, no element, and a
press_key with no key. A model imitates the shape it is shown, so it went on
calling it that way; a real desktop run failed six of eleven calls on shapes
copied from its own history.

This adds computerUseModelCallArgs: the same privacy rule, in the names the
tool accepts, and wires it at tool-runtime.ts, which is the line every
shipping backend goes through. The permission prompt still reads
permissionArgs and the approval scope key is still computed from the raw
call, so only what is written down changes.

What now crosses the privacy boundary

computer-use-privacy-boundary.test.ts asserted this line before this PR, and
it is updated in place rather than around, so the change is visible in the
diff. Two things cross that did not:

  • element_id — an index the model chose into an observation it had already
    read (e12). Admitted only when it is a stable identifier, so an
    accessibility label arriving under that key is dropped rather than persisted.
    Without it, ten element actions in a turn are indistinguishable.
  • coordinate, start_coordinate, region — geometry the model wrote into
    the call itself. Integers only, so a mistyped value still degrades to a
    shape. A coordinate is not read off the screen; withholding it leaves a model
    that clicked a point and missed unable to see that it had already tried that
    point.

What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed. Those keep their key and
lose their value to <text> — the key has to stay, or the model reads its own
history as a call it never made. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so it is
carried: one argument name meaning six things, and only two of the six come
from outside the model.

element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the model's
element_id against the live observation and attaches it, and permissionArgs
is what this projection runs on.

Also here

COMPUTER_USE_SEMANTIC_ACTIONS replaces a hand-written action list that had
already drifted: window_action was added to the strict union and not to the
list, so every window move, resize and minimise would have summarised as
unknown — in the approval a person reads, and in the record the model reads
back. computer-use-schema-parity.test.ts now compares the two in both
directions.

Relationship to #1888

#1888 reached the same conclusion from the renderer's side and carried
elementId in the approval summary so its labels could tell two element rows
apart. That intent is subsumed here: element_id is carried, in the tool's own
casing. When the two are merged, #1888's computer-action-label.ts should read
window_id and element_id rather than windowId and elementId, and the
elementId case it added to computer-use-privacy-boundary.test.ts is covered
by the projection assertion in this branch.

Not here any more

The executor capability declarations this PR was originally titled for were
removed in 0fcfb46: nothing produced them, so they were a type surface with
no writer. The title and this description are rewritten to match what the
branch actually does.

Verification

The new assertion is on the tool_start event, not on the projection function.
Reverted to computerUseApprovalSummary at tool-runtime.ts and rebuilt, it
fails with exactly the diff quoted above; restored, it passes. A test that
calls computerUseModelCallArgs directly stays green either way, which is how
this survived being fixed once already in PiAgentBackend — a backend the
desktop app never instantiates.

@maka/core and @maka/runtime suites pass; npx biome check clean on the
touched files; node scripts/check-console.mjs passes. Rebased onto current
main, cleanly.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all six findings. The claim in the PR body that nothing behaves differently was wrong, and finding 1 was the reason.

Finding 1, resolution: I scoped the constant to what main's schema accepts rather than stacking the PR on the executor branch. Verified first — main's wire enum at computer-use-tools.ts is list_apps, observe, click_element, set_value, select_text, secondary_action, press_key plus CU_ACTION_TYPES, and none of launch_app, scroll_element, element_sequence or window_action is in it. The four names went into APPROVAL_ACTIONS through COMPUTER_USE_SEMANTIC_ACTIONS, so computerUseApprovalSummary flipped knownAction to true for calls the SDK rejects before the tool runs, and rememberForTurnAllowed could evaluate true for the two that also joined SEMANTIC_ACTIONS. I chose scoping over stacking for two reasons. The constant's stated purpose is to be held against the tool schema, so declaring names the schema does not have is the same drift it exists to prevent, only in the direction nothing was checking. And stacking would make the PR unmergeable until the executor lands, which delays the parts that are correct today. The executor's vocabulary belongs on the branch that also ships the executor.

Finding 2: the promised file did not exist, but a guard did — computer-use-schema-parity.test.ts, added in #1870 after this branch was cut. It only walks from the schemas to the catalog, which catches a missing name and not an extra one, so the widening in finding 1 would have passed it. I added the reverse comparison to that file rather than creating the second one the comment named, and corrected the comment to point at the real file. Negative control: reintroducing the four names turns the new check red and it names all four. There is also an in-file control that runs the same comparator over a fixed catalog, so it keeps proving the comparator can return something on the day the real catalog is correct.

Finding 3: confirmed. Only COMPUTER_USE_SEMANTIC_ACTIONS was exported, so computerUseModelCallArgs was unreachable from @maka/runtime and the whole headline fix was dead code while the harmful widening was the only part that shipped. The five missing symbols are exported now, and the projection is wired into projectPiToolArgs in pi-agent-backend.ts, which writes the persisted tool_call message and the tool_start event. Negative control: reverting that one line turns the pi-agent-backend projection test red with the approval dialect back in the diff.

Finding 4: confirmed and dropped rather than wired. ComputerUseSettings appears in none of AppSettings, UpdateAppSettingsInput, the defaults, the merge or the normalize pass, unlike every peer settings module, and select-backend.ts assembles the CU tools with no enable check anywhere. A type that says enabled: false while the shipped app always has it on is worse than no type. Wiring a real switch needs a UI and a migration and does not belong under a types commit.

Finding 5: confirmed, and it is exactly as described. text is six arguments under one name — the key for press_key, key and hold_key, the element action name for secondary_action, a screen substring for select_text, typed content for type. MODEL_CALL_PLAIN_VALUES is now a map from action to the argument names that are plain for that action. An action the tool does not accept has no entry, so it falls through to shapes rather than to whichever set was used last. Negative control: reverting to the name-keyed set turns two tests red, showing press_key coming back as text: .

Finding 6, partially rejected with evidence. dispatch_refused is gone. It is not a declaration — COMPUTER_USE_ERROR_CODES is a runtime value feeding isComputerUseErrorCode, which decides at cua-driver-result.ts:88 whether a raw string from the driver process is trusted or replaced with capture_failed. Widening that with no producer is a live change. The other five named items are different in kind. messageIsAppTextFree, truncated, actions, subrole and placeholder are optional fields on TypeScript interfaces with no runtime effect, and they are not distinguishable from about ten other unproduced declarations in the same diff that the review accepted — focused, appAlias, obscuringRects, menu, query, CuLaunchedApp, scroll_element, window_action, launchApp. Dropping the named five and keeping those would be arbitrary; dropping all of them empties the branch of its purpose. Declaring the shape an executor returns is what this branch is for, so they stay, and the absent-means-withheld polarity on messageIsAppTextFree is kept as suggested.

Also rebased onto current main, which the branch was one commit behind — the missing commit was #1870, the parity test finding 2 turns out to depend on.

Test delta, measured against a baseline taken on clean origin/main on the same machine. This machine's Node has no node:sqlite, so @maka/runtime carries 64 pre-existing failing test names there. After the change: 2722 tests, zero new failures and zero fixed against that set. @maka/core goes 703/703 to 710/710, the seven new ones being the model-call projection tests. biome check clean on the six touched files, check-console passes.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Second-round review addressed. Two findings fixed, one dropped after checking, one scope decision.

Coordinates are back. The doc said typed text, written values and coordinates are screen-derived and stay out, and that is false for the third one: a coordinate is the model's own output, four digits it chose and sent. Reduced to <point>, a model that clicked [412, 88] and missed could not tell whether it had already tried that point, which is the repeated-call shape this projection exists to make visible. coordinate, start_coordinate and region now come back whole when they are integers and still degrade to a shape when they are not, so a malformed argument does not become a leak channel.

The out-of-enum action is reported as sent. On the AI SDK path the question is moot — a schema-rejected call never produces a tool-call part, so settleToolCall is never reached and no tool_start is emitted. But this projection's only consumer is the Pi backend, where tool_start arrives as a transport frame that Maka does not validate, so the case is live there. It is also wrong in principle: this record is the model's own arguments, and the action is one of them. Collapsing an unrecognised name is computerUseApprovalSummary's job, where knownAction decides what a person is asked to allow; a test now pins that the two functions disagree on purpose. The action is bounded and redacted like every other model-supplied string.

The declared-and-unconsumed set is dropped. launchApp, CuLaunchedApp, truncated, actions, subrole, placeholder, focused, menu, query, appAlias, obscuringRects, targetWindowId, targetStacking, messageIsAppTextFree, and the scroll_element and window_action members of CuSemanticAction — zero producers and zero consumers, confirmed by grep across packages and apps. The reason is the parity guard's own header: window_action sat in the strict union and not on the wire, was unreachable from the day it shipped, and nothing failed, because a declaration with no producer has nothing to check it. This branch was putting window_action back into that state with scroll_element beside it. computer-use-types.ts is now byte-identical to main; the additions come with the executor that produces them, where a guard can assert they arrive. The PR body has been rewritten so it no longer claims them.

What is left is what has a consumer: COMPUTER_USE_SEMANTIC_ACTIONS with the two-way parity check, and computerUseModelCallArgs wired into the Pi backend.

Negative controls, run rather than asserted. Reverting the geometry branch turned the coordinate tests red on coordinate: '<point>' against [412, 88]; restoring the APPROVAL_ACTIONS gate on action turned the third red on unknown against element_sequence, and the Pi backend's own projection test with it. Both back to green after restoring.

One thing I did not change, because it is outside this branch: tool-runtime.ts:776 still writes computerUseApprovalSummary(permissionArgs) into both the tool_start event and the persisted tool_call message on the AI SDK path. That is the same defect this PR fixes, on the main backend rather than the Pi one. It needs its own change, since the same value also feeds describeToolIntent, the durable args hash and the approval record.

Test delta against a clean rebuild of main on this machine: none in packages/core or the Computer Use tests. The rest of the run is shell, filesystem and host-process timing under load 40 on this box; each one passes when its file is run alone.

@hqhq1025hqhq1025 changed the title feat(cu): declare the capabilities an executor can have, and the evidence it returnsfix(cu): let a model read back the Computer Use call it actually madeAug 3, 2026
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 0fcfb46 to 03fe3d9CompareAugust 3, 2026 11:11
@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Blocking finding addressed: the fix is moved to the path the shipping app runs.

It was wired into PiAgentBackend, which the desktop app never instantiates.
boot.ts registers only ai-sdk and fake; the pi backend is reachable only from
harbor-cell.ts behind MAKA_PI_*, driving an external CLI, and it does not
construct a ToolRuntime at all. tool-runtime.ts had a zero-byte diff, so on the
shipping path nothing changed. Measured on the real ToolRuntime, a model that
sent action press_key with app, window_id 41, observation_id obs-7, element_id
e12 and text cmd+s read back action, approvalClass, rememberForTurnAllowed,
app, windowId 41, observationId — a key the tool rejects, two fields it never
sent, no element, and a press_key with no key. That is now the one line the PR
changes in production.

The new assertion is on the tool_start event, which is what model-history.ts
replays, rather than on the projection function. Reverted to
computerUseApprovalSummary and rebuilt, it fails with exactly the diff above;
restored, it passes. A test calling computerUseModelCallArgs directly stays
green either way, which is how this survived being fixed once already.

On the coupling with #1888 and the privacy boundary, since the description
asked that it not be resolved silently.

computer-use-privacy-boundary.test.ts is not new to #1888; it has asserted this
line since #1581, and #1888 added an elementId case to it. Both PRs are
therefore changing the same invariant, and this one changes it in place rather
than around it, so what crosses is visible in the diff and is spelled out in
the file, the commit message and the PR body.

Two things now cross. element_id, which is an index the model chose into an
observation it had already read, admitted only when it is a stable identifier
so an accessibility label arriving under that key is dropped — this is exactly
what #1888 wanted and it is subsumed here, in the tool's own casing.
coordinate, start_coordinate and region, which are geometry the model wrote
into the call itself, integers only so a mistyped value still degrades to a
shape.

The argument for carrying coordinates is that they are the model's own output,
already present in the assistant turn that produced them, and that withholding
them leaves a model that clicked a point and missed unable to see that it had
already tried that point. The argument against, that for set_value and type
these are what the user was having typed on their behalf, does not apply to
coordinates: it applies to text and value, and those still do not cross. They
keep their key and lose their value to , because the key has to stay or
the model reads its own history as a call it never made. text for press_key,
key, hold_key and secondary_action is a name from a closed set the executor
publishes, so it is carried; that is one argument name meaning six things, and
only two of the six come from outside the model.

element_identity is added to the host-only set. The Computer Use tool's own
permissionArgs resolves the model's element_id against the live observation and
attaches its identity, and permissionArgs is what this projection now runs on,
so without that the model would read back a key it cannot send whose value came
off the accessibility tree.

For #1888 specifically: when the two are merged, computer-action-label.ts
should read window_id and element_id rather than windowId and elementId, since
the persisted args are now in the tool's dialect. Nothing on main reads the
persisted Computer Use args in the approval-summary dialect — the permission
prompt reads permissionArgs and the approval scope key is computed from the raw
call — so that renderer is the only reader affected.

Non-blocking finding: the title and body are rewritten. 0fcfb46 removed the
capability declarations, so the branch no longer does what it was named for;
what remains is the model-facing projection and the action-list parity guard,
and the description now says so and says what was dropped and why.

Rebased onto current main, cleanly. @maka/core and @maka/runtime suites pass,
npx biome check is clean on the touched files, and node scripts/check-console.mjs
passes.

…ence it returns
Types only. Nothing selects them, nothing behaves differently, and every
existing test passes unchanged — this is the vocabulary a second executor
backend needs before it can be written against the same interface.
What it adds:
- `launchApp` on `CuDispatchBackend`. Starting an application is something an
executor can do and the host could not ask for, so a model that needed a
closed app had no move but to tell the user to open it.
- `truncated` on `CuObservation` and `actions` on `CuObservedElement`. A tree
that was cut short and an element that offers more than a press are both
facts the executor knows and the host had no field to carry, so they were
read, validated and then dropped.
- `COMPUTER_USE_SEMANTIC_ACTIONS`, so the set of element actions has one
written-down list rather than being spelled again at each place that
switches on it.
Optional throughout: a backend that provides none of it satisfies the
interface exactly as before.
… accepts
The previous version of this branch said nothing behaved differently. Four
things did.
`COMPUTER_USE_SEMANTIC_ACTIONS` declared `launch_app`, `scroll_element`,
`element_sequence` and `window_action`. None of those is in the tool's wire
enum on main; they exist only on the branch that adds the executor. The
constant feeds `APPROVAL_ACTIONS`, so `computerUseApprovalSummary` flipped
`knownAction` to true for four names the SDK rejects before the tool ever
runs, and `rememberForTurnAllowed` could evaluate true for two of them. A
person would have read an approval for an action that did not happen. The
constant is now scoped to what this schema accepts. Declaring an executor's
vocabulary is worth doing on the branch that also ships the executor.
`dispatch_refused` had no producer, and adding it to `COMPUTER_USE_ERROR_CODES`
widened `isComputerUseErrorCode` — the filter that decides whether a raw string
from the driver process is passed through or replaced with `capture_failed`.
That is a live trust boundary, not a declaration. Removed until something
produces it. The unproduced fields on the observation and outcome types are
left as they are: they have no runtime effect, and declaring the shape the
executor returns is what this branch is for.
The drift guard the constant's rationale rests on did not exist. The comment
promised a file that was never written, and the guard on main only walks from
the schemas to the catalog, which is the direction that catches a missing name
— not the direction this branch broke. `computer-use-schema-parity.test.ts`
now compares the two in both directions, with a negative control that runs the
real comparator over a fixed catalog so it keeps proving something on the day
the real one is correct.
`ComputerUseSettings` declared `enabled: false` and was registered nowhere:
not in `AppSettings`, not in `UpdateAppSettingsInput`, not in the defaults,
merge or normalize passes that every peer settings module goes through. In the
shipped app Computer Use tools are assembled with no enable check at all, so
the type told a reader the capability is off by default while it is always on.
Dropped. Wiring a real switch is a change with a UI and a migration, and it
does not belong under a types commit.
The model-call projection could not be used. Only `COMPUTER_USE_SEMANTIC_ACTIONS`
was exported from the package entry, so `computerUseModelCallArgs` — the
headline fix — was unreachable from `@maka/runtime`, and the defective
projection it exists to replace stayed in place. It is exported now and wired
into `projectPiToolArgs`, which is what writes the persisted `tool_call`
message and the `tool_start` event: the record the model reads back as its own
call. It had been reading the host's approval summary, which renames
`window_id` to `windowId`, adds two fields the model never sent, and drops
every argument it does not need.
And the projection withheld the wrong half of `text`. It keyed on argument
name, but `text` is six arguments under one name: the key for `press_key`,
`key` and `hold_key`, the element action name for `secondary_action`, a
substring of the window for `select_text`, and typed content for `type`.
Excluding all six is right for two of them. For the rest the model was reading
back `press_key ... text: <text>` and could not see which key it had pressed —
the exact failure the comment cited as motivation. It now keys on the action.
An action the tool does not accept has no plain set, so it falls through to
shapes rather than to whichever set was used last.
…ations nothing produces
The projection withheld coordinates on the grounds that they are
screen-derived. They are not: a coordinate is the model's own output,
four digits it chose and sent. Reduced to `<point>`, a model that clicked
[412, 88] and missed could not tell whether it had already tried that
point — the repeated-call shape this projection exists to make visible,
reintroduced by the projection itself. `coordinate`, `start_coordinate`
and `region` now come back whole when they are integers, and still
degrade to a shape when they are not.
An action the schema does not carry was recorded as `unknown`, so a model
whose call was rejected for the name it sent read its own history as a
call it never made and could not connect the rejection to the cause.
Collapsing an unrecognised name is `computerUseApprovalSummary`'s job,
where `knownAction` decides what a person is asked to allow; this record
is the model's own arguments, and the action is one of them. It is
bounded and redacted like every other model-supplied string.
The rest of the branch declared fourteen fields, one interface, one
backend method and two union members that nothing in the repository sets
and nothing reads. The parity guard added here covers the action-name
catalog and nothing else, so those declarations had nothing holding them
honest — which is the same condition the guard's own header describes:
`window_action` sat in the strict union and not on the wire and was
unreachable from the day it shipped, and nothing failed. This branch was
putting `window_action` back in that state, along with `scroll_element`.
They come with the executor that produces them, where a guard can check
that they arrive.
What is left is what has a consumer: `COMPUTER_USE_SEMANTIC_ACTIONS` with
the two-way parity check that keeps it and the wire enum in step, and
`computerUseModelCallArgs` wired into the Pi backend's tool_call record.
The projection was wired into PiAgentBackend, which the desktop app never
instantiates: boot.ts registers only ai-sdk and fake, and the pi backend
is reachable only from harbor-cell.ts behind MAKA_PI_*, driving an
external CLI, and it does not build a ToolRuntime at all. tool-runtime.ts
had a zero-byte diff, so on the shipping path nothing changed.
Measured on the real ToolRuntime before this commit, for a model that
sent {action:'press_key', app:'TextEdit', window_id:41,
observation_id:'obs-7', element_id:'e12', text:'cmd+s'}, the persisted
tool_start args were {action, approvalClass, rememberForTurnAllowed, app,
windowId:41, observationId} — a key the tool rejects, two host fields the
model never sent, no element, and a press_key with no key. model-history
replays event.content.args, so that is what the model read back of its
own call, and it went on calling it that way.
The line moves; the projection does not. The permission prompt still
reads permissionArgs and the approval scope key is still computed from
the raw call, so only what is written down changes.
Two things now cross the privacy boundary that did not, and
computer-use-privacy-boundary.test.ts says so where it asserts them.
element_id: an index the model chose into an observation it had already
read. Admitted only when it is a stable identifier, so an accessibility
label arriving under that key is dropped rather than persisted. Without
it every element action in a turn reads the same.
coordinate, start_coordinate, region: geometry the model wrote into the
call itself, integers only, so a mistyped value still degrades to a
shape. A coordinate is not read off the screen; withholding it left a
model that clicked a point and missed unable to see that it had already
tried that point.
What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed — those keep their key
and lose their value to <text>. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so
it is carried; that is one argument name meaning six things, and only two
of the six come from outside the model.
element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the
model's element_id against the live observation and attaches it, and
permissionArgs is what this projection now runs on, so without that the
model would read back a key it cannot send whose value came off the
accessibility tree.
The new assertion is on the tool_start event, not on the projection
function. A test that calls computerUseModelCallArgs directly stays green
while the runtime writes the approval summary, which is how this survived
being fixed once already.
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 03fe3d9 to 71df4c1CompareAugust 3, 2026 11:48
@hqhq1025
hqhq1025 merged commit 80eb37d into apache:mainAug 3, 2026
10 checks passed
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit that referenced this pull request Aug 3, 2026
…1888)
* feat(ui): give driving the computer its own row, and say what it did
Computer Use's display name is a noun — "Maka Computer" — so a turn that
observed a window, clicked a button and observed again rendered three
identical rows. `computerActionLabel` derives a row label from the call's
own arguments instead: 观察「计算器」窗口, 点击元素 e7.
The label is derived, never declared. The model is not given an `intent`
field to write; every word it produces is owned by the runtime, and a free
text field would be one more place it can be wrong.
What the renderer can read is `computerUseApprovalSummary(...)`, which
`ToolRuntime.executeTool` substitutes for the wire call before anything is
persisted. `element_id` had to be carried on that projection for a row to
name its target at all, and this adds it — but a shape filter is not a
privacy boundary. `stableIdentifier` admits `[A-Za-z0-9._:-]{1,256}`, which
is also the shape of an API key, and arguments are not validated before the
projection runs, so a model that put a token under `element_id` wrote it
verbatim into the persisted `tool_call`, onto `tool_start` and into the row
— while the same string under `observation_id` came out `[redacted]`. It
now takes the same `redactSecrets` pass `observationId` and `app` do.
Also puts `'computer'` on the wire as a tool activity kind and has the
Session decoder read the one list rather than a second hand-written copy of
it, which had already fallen behind and rejected the new kind outright.
Note for clients behind the runtime-host Session protocol: the `tool_start`
frame carries no `args`, so such a client gets the kind but cannot derive
the label. The desktop receives full SessionEvents over IPC and is
unaffected.
* fix(ui): read the Computer Use row's arguments in the dialect they are now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. #1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after #1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hqhq1025
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' fix(cu): let a model read back the Computer Use call it actually made by hqhq1025 · Pull Request #1891 · apache/maka · GitHub
Skip to content

fix(cu): let a model read back the Computer Use call it actually made - #1891

Merged
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types
Aug 3, 2026
Merged

fix(cu): let a model read back the Computer Use call it actually made#1891
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types

Conversation

@hqhq1025

@hqhq1025hqhq1025 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The record a model reads back of its own Computer Use call was the host's
approval projection, not the call. model-history.ts replays
event.content.args, and those args came from computerUseApprovalSummary,
which exists to decide and display a permission: it renames window_id to
windowId, adds approvalClass and rememberForTurnAllowed, and drops every
argument it does not need.

Measured on the real ToolRuntime, a model that sent

{"action":"press_key","app":"TextEdit","window_id":41,
"observation_id":"obs-7","element_id":"e12","text":"cmd+s"}

read back

{"action":"press_key","approvalClass":"keyboard_mutation",
"rememberForTurnAllowed":true,"app":"TextEdit","windowId":41,
"observationId":"obs-7"}

— a key the tool rejects, two fields it never sent, no element, and a
press_key with no key. A model imitates the shape it is shown, so it went on
calling it that way; a real desktop run failed six of eleven calls on shapes
copied from its own history.

This adds computerUseModelCallArgs: the same privacy rule, in the names the
tool accepts, and wires it at tool-runtime.ts, which is the line every
shipping backend goes through. The permission prompt still reads
permissionArgs and the approval scope key is still computed from the raw
call, so only what is written down changes.

What now crosses the privacy boundary

computer-use-privacy-boundary.test.ts asserted this line before this PR, and
it is updated in place rather than around, so the change is visible in the
diff. Two things cross that did not:

  • element_id — an index the model chose into an observation it had already
    read (e12). Admitted only when it is a stable identifier, so an
    accessibility label arriving under that key is dropped rather than persisted.
    Without it, ten element actions in a turn are indistinguishable.
  • coordinate, start_coordinate, region — geometry the model wrote into
    the call itself. Integers only, so a mistyped value still degrades to a
    shape. A coordinate is not read off the screen; withholding it leaves a model
    that clicked a point and missed unable to see that it had already tried that
    point.

What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed. Those keep their key and
lose their value to <text> — the key has to stay, or the model reads its own
history as a call it never made. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so it is
carried: one argument name meaning six things, and only two of the six come
from outside the model.

element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the model's
element_id against the live observation and attaches it, and permissionArgs
is what this projection runs on.

Also here

COMPUTER_USE_SEMANTIC_ACTIONS replaces a hand-written action list that had
already drifted: window_action was added to the strict union and not to the
list, so every window move, resize and minimise would have summarised as
unknown — in the approval a person reads, and in the record the model reads
back. computer-use-schema-parity.test.ts now compares the two in both
directions.

Relationship to #1888

#1888 reached the same conclusion from the renderer's side and carried
elementId in the approval summary so its labels could tell two element rows
apart. That intent is subsumed here: element_id is carried, in the tool's own
casing. When the two are merged, #1888's computer-action-label.ts should read
window_id and element_id rather than windowId and elementId, and the
elementId case it added to computer-use-privacy-boundary.test.ts is covered
by the projection assertion in this branch.

Not here any more

The executor capability declarations this PR was originally titled for were
removed in 0fcfb46: nothing produced them, so they were a type surface with
no writer. The title and this description are rewritten to match what the
branch actually does.

Verification

The new assertion is on the tool_start event, not on the projection function.
Reverted to computerUseApprovalSummary at tool-runtime.ts and rebuilt, it
fails with exactly the diff quoted above; restored, it passes. A test that
calls computerUseModelCallArgs directly stays green either way, which is how
this survived being fixed once already in PiAgentBackend — a backend the
desktop app never instantiates.

@maka/core and @maka/runtime suites pass; npx biome check clean on the
touched files; node scripts/check-console.mjs passes. Rebased onto current
main, cleanly.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all six findings. The claim in the PR body that nothing behaves differently was wrong, and finding 1 was the reason.

Finding 1, resolution: I scoped the constant to what main's schema accepts rather than stacking the PR on the executor branch. Verified first — main's wire enum at computer-use-tools.ts is list_apps, observe, click_element, set_value, select_text, secondary_action, press_key plus CU_ACTION_TYPES, and none of launch_app, scroll_element, element_sequence or window_action is in it. The four names went into APPROVAL_ACTIONS through COMPUTER_USE_SEMANTIC_ACTIONS, so computerUseApprovalSummary flipped knownAction to true for calls the SDK rejects before the tool runs, and rememberForTurnAllowed could evaluate true for the two that also joined SEMANTIC_ACTIONS. I chose scoping over stacking for two reasons. The constant's stated purpose is to be held against the tool schema, so declaring names the schema does not have is the same drift it exists to prevent, only in the direction nothing was checking. And stacking would make the PR unmergeable until the executor lands, which delays the parts that are correct today. The executor's vocabulary belongs on the branch that also ships the executor.

Finding 2: the promised file did not exist, but a guard did — computer-use-schema-parity.test.ts, added in #1870 after this branch was cut. It only walks from the schemas to the catalog, which catches a missing name and not an extra one, so the widening in finding 1 would have passed it. I added the reverse comparison to that file rather than creating the second one the comment named, and corrected the comment to point at the real file. Negative control: reintroducing the four names turns the new check red and it names all four. There is also an in-file control that runs the same comparator over a fixed catalog, so it keeps proving the comparator can return something on the day the real catalog is correct.

Finding 3: confirmed. Only COMPUTER_USE_SEMANTIC_ACTIONS was exported, so computerUseModelCallArgs was unreachable from @maka/runtime and the whole headline fix was dead code while the harmful widening was the only part that shipped. The five missing symbols are exported now, and the projection is wired into projectPiToolArgs in pi-agent-backend.ts, which writes the persisted tool_call message and the tool_start event. Negative control: reverting that one line turns the pi-agent-backend projection test red with the approval dialect back in the diff.

Finding 4: confirmed and dropped rather than wired. ComputerUseSettings appears in none of AppSettings, UpdateAppSettingsInput, the defaults, the merge or the normalize pass, unlike every peer settings module, and select-backend.ts assembles the CU tools with no enable check anywhere. A type that says enabled: false while the shipped app always has it on is worse than no type. Wiring a real switch needs a UI and a migration and does not belong under a types commit.

Finding 5: confirmed, and it is exactly as described. text is six arguments under one name — the key for press_key, key and hold_key, the element action name for secondary_action, a screen substring for select_text, typed content for type. MODEL_CALL_PLAIN_VALUES is now a map from action to the argument names that are plain for that action. An action the tool does not accept has no entry, so it falls through to shapes rather than to whichever set was used last. Negative control: reverting to the name-keyed set turns two tests red, showing press_key coming back as text: .

Finding 6, partially rejected with evidence. dispatch_refused is gone. It is not a declaration — COMPUTER_USE_ERROR_CODES is a runtime value feeding isComputerUseErrorCode, which decides at cua-driver-result.ts:88 whether a raw string from the driver process is trusted or replaced with capture_failed. Widening that with no producer is a live change. The other five named items are different in kind. messageIsAppTextFree, truncated, actions, subrole and placeholder are optional fields on TypeScript interfaces with no runtime effect, and they are not distinguishable from about ten other unproduced declarations in the same diff that the review accepted — focused, appAlias, obscuringRects, menu, query, CuLaunchedApp, scroll_element, window_action, launchApp. Dropping the named five and keeping those would be arbitrary; dropping all of them empties the branch of its purpose. Declaring the shape an executor returns is what this branch is for, so they stay, and the absent-means-withheld polarity on messageIsAppTextFree is kept as suggested.

Also rebased onto current main, which the branch was one commit behind — the missing commit was #1870, the parity test finding 2 turns out to depend on.

Test delta, measured against a baseline taken on clean origin/main on the same machine. This machine's Node has no node:sqlite, so @maka/runtime carries 64 pre-existing failing test names there. After the change: 2722 tests, zero new failures and zero fixed against that set. @maka/core goes 703/703 to 710/710, the seven new ones being the model-call projection tests. biome check clean on the six touched files, check-console passes.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Second-round review addressed. Two findings fixed, one dropped after checking, one scope decision.

Coordinates are back. The doc said typed text, written values and coordinates are screen-derived and stay out, and that is false for the third one: a coordinate is the model's own output, four digits it chose and sent. Reduced to <point>, a model that clicked [412, 88] and missed could not tell whether it had already tried that point, which is the repeated-call shape this projection exists to make visible. coordinate, start_coordinate and region now come back whole when they are integers and still degrade to a shape when they are not, so a malformed argument does not become a leak channel.

The out-of-enum action is reported as sent. On the AI SDK path the question is moot — a schema-rejected call never produces a tool-call part, so settleToolCall is never reached and no tool_start is emitted. But this projection's only consumer is the Pi backend, where tool_start arrives as a transport frame that Maka does not validate, so the case is live there. It is also wrong in principle: this record is the model's own arguments, and the action is one of them. Collapsing an unrecognised name is computerUseApprovalSummary's job, where knownAction decides what a person is asked to allow; a test now pins that the two functions disagree on purpose. The action is bounded and redacted like every other model-supplied string.

The declared-and-unconsumed set is dropped. launchApp, CuLaunchedApp, truncated, actions, subrole, placeholder, focused, menu, query, appAlias, obscuringRects, targetWindowId, targetStacking, messageIsAppTextFree, and the scroll_element and window_action members of CuSemanticAction — zero producers and zero consumers, confirmed by grep across packages and apps. The reason is the parity guard's own header: window_action sat in the strict union and not on the wire, was unreachable from the day it shipped, and nothing failed, because a declaration with no producer has nothing to check it. This branch was putting window_action back into that state with scroll_element beside it. computer-use-types.ts is now byte-identical to main; the additions come with the executor that produces them, where a guard can assert they arrive. The PR body has been rewritten so it no longer claims them.

What is left is what has a consumer: COMPUTER_USE_SEMANTIC_ACTIONS with the two-way parity check, and computerUseModelCallArgs wired into the Pi backend.

Negative controls, run rather than asserted. Reverting the geometry branch turned the coordinate tests red on coordinate: '<point>' against [412, 88]; restoring the APPROVAL_ACTIONS gate on action turned the third red on unknown against element_sequence, and the Pi backend's own projection test with it. Both back to green after restoring.

One thing I did not change, because it is outside this branch: tool-runtime.ts:776 still writes computerUseApprovalSummary(permissionArgs) into both the tool_start event and the persisted tool_call message on the AI SDK path. That is the same defect this PR fixes, on the main backend rather than the Pi one. It needs its own change, since the same value also feeds describeToolIntent, the durable args hash and the approval record.

Test delta against a clean rebuild of main on this machine: none in packages/core or the Computer Use tests. The rest of the run is shell, filesystem and host-process timing under load 40 on this box; each one passes when its file is run alone.

@hqhq1025hqhq1025 changed the title feat(cu): declare the capabilities an executor can have, and the evidence it returnsfix(cu): let a model read back the Computer Use call it actually madeAug 3, 2026
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 0fcfb46 to 03fe3d9CompareAugust 3, 2026 11:11
@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Blocking finding addressed: the fix is moved to the path the shipping app runs.

It was wired into PiAgentBackend, which the desktop app never instantiates.
boot.ts registers only ai-sdk and fake; the pi backend is reachable only from
harbor-cell.ts behind MAKA_PI_*, driving an external CLI, and it does not
construct a ToolRuntime at all. tool-runtime.ts had a zero-byte diff, so on the
shipping path nothing changed. Measured on the real ToolRuntime, a model that
sent action press_key with app, window_id 41, observation_id obs-7, element_id
e12 and text cmd+s read back action, approvalClass, rememberForTurnAllowed,
app, windowId 41, observationId — a key the tool rejects, two fields it never
sent, no element, and a press_key with no key. That is now the one line the PR
changes in production.

The new assertion is on the tool_start event, which is what model-history.ts
replays, rather than on the projection function. Reverted to
computerUseApprovalSummary and rebuilt, it fails with exactly the diff above;
restored, it passes. A test calling computerUseModelCallArgs directly stays
green either way, which is how this survived being fixed once already.

On the coupling with #1888 and the privacy boundary, since the description
asked that it not be resolved silently.

computer-use-privacy-boundary.test.ts is not new to #1888; it has asserted this
line since #1581, and #1888 added an elementId case to it. Both PRs are
therefore changing the same invariant, and this one changes it in place rather
than around it, so what crosses is visible in the diff and is spelled out in
the file, the commit message and the PR body.

Two things now cross. element_id, which is an index the model chose into an
observation it had already read, admitted only when it is a stable identifier
so an accessibility label arriving under that key is dropped — this is exactly
what #1888 wanted and it is subsumed here, in the tool's own casing.
coordinate, start_coordinate and region, which are geometry the model wrote
into the call itself, integers only so a mistyped value still degrades to a
shape.

The argument for carrying coordinates is that they are the model's own output,
already present in the assistant turn that produced them, and that withholding
them leaves a model that clicked a point and missed unable to see that it had
already tried that point. The argument against, that for set_value and type
these are what the user was having typed on their behalf, does not apply to
coordinates: it applies to text and value, and those still do not cross. They
keep their key and lose their value to , because the key has to stay or
the model reads its own history as a call it never made. text for press_key,
key, hold_key and secondary_action is a name from a closed set the executor
publishes, so it is carried; that is one argument name meaning six things, and
only two of the six come from outside the model.

element_identity is added to the host-only set. The Computer Use tool's own
permissionArgs resolves the model's element_id against the live observation and
attaches its identity, and permissionArgs is what this projection now runs on,
so without that the model would read back a key it cannot send whose value came
off the accessibility tree.

For #1888 specifically: when the two are merged, computer-action-label.ts
should read window_id and element_id rather than windowId and elementId, since
the persisted args are now in the tool's dialect. Nothing on main reads the
persisted Computer Use args in the approval-summary dialect — the permission
prompt reads permissionArgs and the approval scope key is computed from the raw
call — so that renderer is the only reader affected.

Non-blocking finding: the title and body are rewritten. 0fcfb46 removed the
capability declarations, so the branch no longer does what it was named for;
what remains is the model-facing projection and the action-list parity guard,
and the description now says so and says what was dropped and why.

Rebased onto current main, cleanly. @maka/core and @maka/runtime suites pass,
npx biome check is clean on the touched files, and node scripts/check-console.mjs
passes.

…ence it returns
Types only. Nothing selects them, nothing behaves differently, and every
existing test passes unchanged — this is the vocabulary a second executor
backend needs before it can be written against the same interface.
What it adds:
- `launchApp` on `CuDispatchBackend`. Starting an application is something an
executor can do and the host could not ask for, so a model that needed a
closed app had no move but to tell the user to open it.
- `truncated` on `CuObservation` and `actions` on `CuObservedElement`. A tree
that was cut short and an element that offers more than a press are both
facts the executor knows and the host had no field to carry, so they were
read, validated and then dropped.
- `COMPUTER_USE_SEMANTIC_ACTIONS`, so the set of element actions has one
written-down list rather than being spelled again at each place that
switches on it.
Optional throughout: a backend that provides none of it satisfies the
interface exactly as before.
… accepts
The previous version of this branch said nothing behaved differently. Four
things did.
`COMPUTER_USE_SEMANTIC_ACTIONS` declared `launch_app`, `scroll_element`,
`element_sequence` and `window_action`. None of those is in the tool's wire
enum on main; they exist only on the branch that adds the executor. The
constant feeds `APPROVAL_ACTIONS`, so `computerUseApprovalSummary` flipped
`knownAction` to true for four names the SDK rejects before the tool ever
runs, and `rememberForTurnAllowed` could evaluate true for two of them. A
person would have read an approval for an action that did not happen. The
constant is now scoped to what this schema accepts. Declaring an executor's
vocabulary is worth doing on the branch that also ships the executor.
`dispatch_refused` had no producer, and adding it to `COMPUTER_USE_ERROR_CODES`
widened `isComputerUseErrorCode` — the filter that decides whether a raw string
from the driver process is passed through or replaced with `capture_failed`.
That is a live trust boundary, not a declaration. Removed until something
produces it. The unproduced fields on the observation and outcome types are
left as they are: they have no runtime effect, and declaring the shape the
executor returns is what this branch is for.
The drift guard the constant's rationale rests on did not exist. The comment
promised a file that was never written, and the guard on main only walks from
the schemas to the catalog, which is the direction that catches a missing name
— not the direction this branch broke. `computer-use-schema-parity.test.ts`
now compares the two in both directions, with a negative control that runs the
real comparator over a fixed catalog so it keeps proving something on the day
the real one is correct.
`ComputerUseSettings` declared `enabled: false` and was registered nowhere:
not in `AppSettings`, not in `UpdateAppSettingsInput`, not in the defaults,
merge or normalize passes that every peer settings module goes through. In the
shipped app Computer Use tools are assembled with no enable check at all, so
the type told a reader the capability is off by default while it is always on.
Dropped. Wiring a real switch is a change with a UI and a migration, and it
does not belong under a types commit.
The model-call projection could not be used. Only `COMPUTER_USE_SEMANTIC_ACTIONS`
was exported from the package entry, so `computerUseModelCallArgs` — the
headline fix — was unreachable from `@maka/runtime`, and the defective
projection it exists to replace stayed in place. It is exported now and wired
into `projectPiToolArgs`, which is what writes the persisted `tool_call`
message and the `tool_start` event: the record the model reads back as its own
call. It had been reading the host's approval summary, which renames
`window_id` to `windowId`, adds two fields the model never sent, and drops
every argument it does not need.
And the projection withheld the wrong half of `text`. It keyed on argument
name, but `text` is six arguments under one name: the key for `press_key`,
`key` and `hold_key`, the element action name for `secondary_action`, a
substring of the window for `select_text`, and typed content for `type`.
Excluding all six is right for two of them. For the rest the model was reading
back `press_key ... text: <text>` and could not see which key it had pressed —
the exact failure the comment cited as motivation. It now keys on the action.
An action the tool does not accept has no plain set, so it falls through to
shapes rather than to whichever set was used last.
…ations nothing produces
The projection withheld coordinates on the grounds that they are
screen-derived. They are not: a coordinate is the model's own output,
four digits it chose and sent. Reduced to `<point>`, a model that clicked
[412, 88] and missed could not tell whether it had already tried that
point — the repeated-call shape this projection exists to make visible,
reintroduced by the projection itself. `coordinate`, `start_coordinate`
and `region` now come back whole when they are integers, and still
degrade to a shape when they are not.
An action the schema does not carry was recorded as `unknown`, so a model
whose call was rejected for the name it sent read its own history as a
call it never made and could not connect the rejection to the cause.
Collapsing an unrecognised name is `computerUseApprovalSummary`'s job,
where `knownAction` decides what a person is asked to allow; this record
is the model's own arguments, and the action is one of them. It is
bounded and redacted like every other model-supplied string.
The rest of the branch declared fourteen fields, one interface, one
backend method and two union members that nothing in the repository sets
and nothing reads. The parity guard added here covers the action-name
catalog and nothing else, so those declarations had nothing holding them
honest — which is the same condition the guard's own header describes:
`window_action` sat in the strict union and not on the wire and was
unreachable from the day it shipped, and nothing failed. This branch was
putting `window_action` back in that state, along with `scroll_element`.
They come with the executor that produces them, where a guard can check
that they arrive.
What is left is what has a consumer: `COMPUTER_USE_SEMANTIC_ACTIONS` with
the two-way parity check that keeps it and the wire enum in step, and
`computerUseModelCallArgs` wired into the Pi backend's tool_call record.
The projection was wired into PiAgentBackend, which the desktop app never
instantiates: boot.ts registers only ai-sdk and fake, and the pi backend
is reachable only from harbor-cell.ts behind MAKA_PI_*, driving an
external CLI, and it does not build a ToolRuntime at all. tool-runtime.ts
had a zero-byte diff, so on the shipping path nothing changed.
Measured on the real ToolRuntime before this commit, for a model that
sent {action:'press_key', app:'TextEdit', window_id:41,
observation_id:'obs-7', element_id:'e12', text:'cmd+s'}, the persisted
tool_start args were {action, approvalClass, rememberForTurnAllowed, app,
windowId:41, observationId} — a key the tool rejects, two host fields the
model never sent, no element, and a press_key with no key. model-history
replays event.content.args, so that is what the model read back of its
own call, and it went on calling it that way.
The line moves; the projection does not. The permission prompt still
reads permissionArgs and the approval scope key is still computed from
the raw call, so only what is written down changes.
Two things now cross the privacy boundary that did not, and
computer-use-privacy-boundary.test.ts says so where it asserts them.
element_id: an index the model chose into an observation it had already
read. Admitted only when it is a stable identifier, so an accessibility
label arriving under that key is dropped rather than persisted. Without
it every element action in a turn reads the same.
coordinate, start_coordinate, region: geometry the model wrote into the
call itself, integers only, so a mistyped value still degrades to a
shape. A coordinate is not read off the screen; withholding it left a
model that clicked a point and missed unable to see that it had already
tried that point.
What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed — those keep their key
and lose their value to <text>. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so
it is carried; that is one argument name meaning six things, and only two
of the six come from outside the model.
element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the
model's element_id against the live observation and attaches it, and
permissionArgs is what this projection now runs on, so without that the
model would read back a key it cannot send whose value came off the
accessibility tree.
The new assertion is on the tool_start event, not on the projection
function. A test that calls computerUseModelCallArgs directly stays green
while the runtime writes the approval summary, which is how this survived
being fixed once already.
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 03fe3d9 to 71df4c1CompareAugust 3, 2026 11:48
@hqhq1025
hqhq1025 merged commit 80eb37d into apache:mainAug 3, 2026
10 checks passed
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit that referenced this pull request Aug 3, 2026
…1888)
* feat(ui): give driving the computer its own row, and say what it did
Computer Use's display name is a noun — "Maka Computer" — so a turn that
observed a window, clicked a button and observed again rendered three
identical rows. `computerActionLabel` derives a row label from the call's
own arguments instead: 观察「计算器」窗口, 点击元素 e7.
The label is derived, never declared. The model is not given an `intent`
field to write; every word it produces is owned by the runtime, and a free
text field would be one more place it can be wrong.
What the renderer can read is `computerUseApprovalSummary(...)`, which
`ToolRuntime.executeTool` substitutes for the wire call before anything is
persisted. `element_id` had to be carried on that projection for a row to
name its target at all, and this adds it — but a shape filter is not a
privacy boundary. `stableIdentifier` admits `[A-Za-z0-9._:-]{1,256}`, which
is also the shape of an API key, and arguments are not validated before the
projection runs, so a model that put a token under `element_id` wrote it
verbatim into the persisted `tool_call`, onto `tool_start` and into the row
— while the same string under `observation_id` came out `[redacted]`. It
now takes the same `redactSecrets` pass `observationId` and `app` do.
Also puts `'computer'` on the wire as a tool activity kind and has the
Session decoder read the one list rather than a second hand-written copy of
it, which had already fallen behind and rejected the new kind outright.
Note for clients behind the runtime-host Session protocol: the `tool_start`
frame carries no `args`, so such a client gets the kind but cannot derive
the label. The desktop receives full SessionEvents over IPC and is
unaffected.
* fix(ui): read the Computer Use row's arguments in the dialect they are now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. #1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after #1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hqhq1025
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(cu): let a model read back the Computer Use call it actually made by hqhq1025 · Pull Request #1891 · apache/maka · GitHub
Skip to content

fix(cu): let a model read back the Computer Use call it actually made - #1891

Merged
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types
Aug 3, 2026
Merged

fix(cu): let a model read back the Computer Use call it actually made#1891
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types

Conversation

@hqhq1025

@hqhq1025hqhq1025 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The record a model reads back of its own Computer Use call was the host's
approval projection, not the call. model-history.ts replays
event.content.args, and those args came from computerUseApprovalSummary,
which exists to decide and display a permission: it renames window_id to
windowId, adds approvalClass and rememberForTurnAllowed, and drops every
argument it does not need.

Measured on the real ToolRuntime, a model that sent

{"action":"press_key","app":"TextEdit","window_id":41,
"observation_id":"obs-7","element_id":"e12","text":"cmd+s"}

read back

{"action":"press_key","approvalClass":"keyboard_mutation",
"rememberForTurnAllowed":true,"app":"TextEdit","windowId":41,
"observationId":"obs-7"}

— a key the tool rejects, two fields it never sent, no element, and a
press_key with no key. A model imitates the shape it is shown, so it went on
calling it that way; a real desktop run failed six of eleven calls on shapes
copied from its own history.

This adds computerUseModelCallArgs: the same privacy rule, in the names the
tool accepts, and wires it at tool-runtime.ts, which is the line every
shipping backend goes through. The permission prompt still reads
permissionArgs and the approval scope key is still computed from the raw
call, so only what is written down changes.

What now crosses the privacy boundary

computer-use-privacy-boundary.test.ts asserted this line before this PR, and
it is updated in place rather than around, so the change is visible in the
diff. Two things cross that did not:

  • element_id — an index the model chose into an observation it had already
    read (e12). Admitted only when it is a stable identifier, so an
    accessibility label arriving under that key is dropped rather than persisted.
    Without it, ten element actions in a turn are indistinguishable.
  • coordinate, start_coordinate, region — geometry the model wrote into
    the call itself. Integers only, so a mistyped value still degrades to a
    shape. A coordinate is not read off the screen; withholding it leaves a model
    that clicked a point and missed unable to see that it had already tried that
    point.

What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed. Those keep their key and
lose their value to <text> — the key has to stay, or the model reads its own
history as a call it never made. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so it is
carried: one argument name meaning six things, and only two of the six come
from outside the model.

element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the model's
element_id against the live observation and attaches it, and permissionArgs
is what this projection runs on.

Also here

COMPUTER_USE_SEMANTIC_ACTIONS replaces a hand-written action list that had
already drifted: window_action was added to the strict union and not to the
list, so every window move, resize and minimise would have summarised as
unknown — in the approval a person reads, and in the record the model reads
back. computer-use-schema-parity.test.ts now compares the two in both
directions.

Relationship to #1888

#1888 reached the same conclusion from the renderer's side and carried
elementId in the approval summary so its labels could tell two element rows
apart. That intent is subsumed here: element_id is carried, in the tool's own
casing. When the two are merged, #1888's computer-action-label.ts should read
window_id and element_id rather than windowId and elementId, and the
elementId case it added to computer-use-privacy-boundary.test.ts is covered
by the projection assertion in this branch.

Not here any more

The executor capability declarations this PR was originally titled for were
removed in 0fcfb46: nothing produced them, so they were a type surface with
no writer. The title and this description are rewritten to match what the
branch actually does.

Verification

The new assertion is on the tool_start event, not on the projection function.
Reverted to computerUseApprovalSummary at tool-runtime.ts and rebuilt, it
fails with exactly the diff quoted above; restored, it passes. A test that
calls computerUseModelCallArgs directly stays green either way, which is how
this survived being fixed once already in PiAgentBackend — a backend the
desktop app never instantiates.

@maka/core and @maka/runtime suites pass; npx biome check clean on the
touched files; node scripts/check-console.mjs passes. Rebased onto current
main, cleanly.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all six findings. The claim in the PR body that nothing behaves differently was wrong, and finding 1 was the reason.

Finding 1, resolution: I scoped the constant to what main's schema accepts rather than stacking the PR on the executor branch. Verified first — main's wire enum at computer-use-tools.ts is list_apps, observe, click_element, set_value, select_text, secondary_action, press_key plus CU_ACTION_TYPES, and none of launch_app, scroll_element, element_sequence or window_action is in it. The four names went into APPROVAL_ACTIONS through COMPUTER_USE_SEMANTIC_ACTIONS, so computerUseApprovalSummary flipped knownAction to true for calls the SDK rejects before the tool runs, and rememberForTurnAllowed could evaluate true for the two that also joined SEMANTIC_ACTIONS. I chose scoping over stacking for two reasons. The constant's stated purpose is to be held against the tool schema, so declaring names the schema does not have is the same drift it exists to prevent, only in the direction nothing was checking. And stacking would make the PR unmergeable until the executor lands, which delays the parts that are correct today. The executor's vocabulary belongs on the branch that also ships the executor.

Finding 2: the promised file did not exist, but a guard did — computer-use-schema-parity.test.ts, added in #1870 after this branch was cut. It only walks from the schemas to the catalog, which catches a missing name and not an extra one, so the widening in finding 1 would have passed it. I added the reverse comparison to that file rather than creating the second one the comment named, and corrected the comment to point at the real file. Negative control: reintroducing the four names turns the new check red and it names all four. There is also an in-file control that runs the same comparator over a fixed catalog, so it keeps proving the comparator can return something on the day the real catalog is correct.

Finding 3: confirmed. Only COMPUTER_USE_SEMANTIC_ACTIONS was exported, so computerUseModelCallArgs was unreachable from @maka/runtime and the whole headline fix was dead code while the harmful widening was the only part that shipped. The five missing symbols are exported now, and the projection is wired into projectPiToolArgs in pi-agent-backend.ts, which writes the persisted tool_call message and the tool_start event. Negative control: reverting that one line turns the pi-agent-backend projection test red with the approval dialect back in the diff.

Finding 4: confirmed and dropped rather than wired. ComputerUseSettings appears in none of AppSettings, UpdateAppSettingsInput, the defaults, the merge or the normalize pass, unlike every peer settings module, and select-backend.ts assembles the CU tools with no enable check anywhere. A type that says enabled: false while the shipped app always has it on is worse than no type. Wiring a real switch needs a UI and a migration and does not belong under a types commit.

Finding 5: confirmed, and it is exactly as described. text is six arguments under one name — the key for press_key, key and hold_key, the element action name for secondary_action, a screen substring for select_text, typed content for type. MODEL_CALL_PLAIN_VALUES is now a map from action to the argument names that are plain for that action. An action the tool does not accept has no entry, so it falls through to shapes rather than to whichever set was used last. Negative control: reverting to the name-keyed set turns two tests red, showing press_key coming back as text: .

Finding 6, partially rejected with evidence. dispatch_refused is gone. It is not a declaration — COMPUTER_USE_ERROR_CODES is a runtime value feeding isComputerUseErrorCode, which decides at cua-driver-result.ts:88 whether a raw string from the driver process is trusted or replaced with capture_failed. Widening that with no producer is a live change. The other five named items are different in kind. messageIsAppTextFree, truncated, actions, subrole and placeholder are optional fields on TypeScript interfaces with no runtime effect, and they are not distinguishable from about ten other unproduced declarations in the same diff that the review accepted — focused, appAlias, obscuringRects, menu, query, CuLaunchedApp, scroll_element, window_action, launchApp. Dropping the named five and keeping those would be arbitrary; dropping all of them empties the branch of its purpose. Declaring the shape an executor returns is what this branch is for, so they stay, and the absent-means-withheld polarity on messageIsAppTextFree is kept as suggested.

Also rebased onto current main, which the branch was one commit behind — the missing commit was #1870, the parity test finding 2 turns out to depend on.

Test delta, measured against a baseline taken on clean origin/main on the same machine. This machine's Node has no node:sqlite, so @maka/runtime carries 64 pre-existing failing test names there. After the change: 2722 tests, zero new failures and zero fixed against that set. @maka/core goes 703/703 to 710/710, the seven new ones being the model-call projection tests. biome check clean on the six touched files, check-console passes.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Second-round review addressed. Two findings fixed, one dropped after checking, one scope decision.

Coordinates are back. The doc said typed text, written values and coordinates are screen-derived and stay out, and that is false for the third one: a coordinate is the model's own output, four digits it chose and sent. Reduced to <point>, a model that clicked [412, 88] and missed could not tell whether it had already tried that point, which is the repeated-call shape this projection exists to make visible. coordinate, start_coordinate and region now come back whole when they are integers and still degrade to a shape when they are not, so a malformed argument does not become a leak channel.

The out-of-enum action is reported as sent. On the AI SDK path the question is moot — a schema-rejected call never produces a tool-call part, so settleToolCall is never reached and no tool_start is emitted. But this projection's only consumer is the Pi backend, where tool_start arrives as a transport frame that Maka does not validate, so the case is live there. It is also wrong in principle: this record is the model's own arguments, and the action is one of them. Collapsing an unrecognised name is computerUseApprovalSummary's job, where knownAction decides what a person is asked to allow; a test now pins that the two functions disagree on purpose. The action is bounded and redacted like every other model-supplied string.

The declared-and-unconsumed set is dropped. launchApp, CuLaunchedApp, truncated, actions, subrole, placeholder, focused, menu, query, appAlias, obscuringRects, targetWindowId, targetStacking, messageIsAppTextFree, and the scroll_element and window_action members of CuSemanticAction — zero producers and zero consumers, confirmed by grep across packages and apps. The reason is the parity guard's own header: window_action sat in the strict union and not on the wire, was unreachable from the day it shipped, and nothing failed, because a declaration with no producer has nothing to check it. This branch was putting window_action back into that state with scroll_element beside it. computer-use-types.ts is now byte-identical to main; the additions come with the executor that produces them, where a guard can assert they arrive. The PR body has been rewritten so it no longer claims them.

What is left is what has a consumer: COMPUTER_USE_SEMANTIC_ACTIONS with the two-way parity check, and computerUseModelCallArgs wired into the Pi backend.

Negative controls, run rather than asserted. Reverting the geometry branch turned the coordinate tests red on coordinate: '<point>' against [412, 88]; restoring the APPROVAL_ACTIONS gate on action turned the third red on unknown against element_sequence, and the Pi backend's own projection test with it. Both back to green after restoring.

One thing I did not change, because it is outside this branch: tool-runtime.ts:776 still writes computerUseApprovalSummary(permissionArgs) into both the tool_start event and the persisted tool_call message on the AI SDK path. That is the same defect this PR fixes, on the main backend rather than the Pi one. It needs its own change, since the same value also feeds describeToolIntent, the durable args hash and the approval record.

Test delta against a clean rebuild of main on this machine: none in packages/core or the Computer Use tests. The rest of the run is shell, filesystem and host-process timing under load 40 on this box; each one passes when its file is run alone.

@hqhq1025hqhq1025 changed the title feat(cu): declare the capabilities an executor can have, and the evidence it returnsfix(cu): let a model read back the Computer Use call it actually madeAug 3, 2026
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 0fcfb46 to 03fe3d9CompareAugust 3, 2026 11:11
@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Blocking finding addressed: the fix is moved to the path the shipping app runs.

It was wired into PiAgentBackend, which the desktop app never instantiates.
boot.ts registers only ai-sdk and fake; the pi backend is reachable only from
harbor-cell.ts behind MAKA_PI_*, driving an external CLI, and it does not
construct a ToolRuntime at all. tool-runtime.ts had a zero-byte diff, so on the
shipping path nothing changed. Measured on the real ToolRuntime, a model that
sent action press_key with app, window_id 41, observation_id obs-7, element_id
e12 and text cmd+s read back action, approvalClass, rememberForTurnAllowed,
app, windowId 41, observationId — a key the tool rejects, two fields it never
sent, no element, and a press_key with no key. That is now the one line the PR
changes in production.

The new assertion is on the tool_start event, which is what model-history.ts
replays, rather than on the projection function. Reverted to
computerUseApprovalSummary and rebuilt, it fails with exactly the diff above;
restored, it passes. A test calling computerUseModelCallArgs directly stays
green either way, which is how this survived being fixed once already.

On the coupling with #1888 and the privacy boundary, since the description
asked that it not be resolved silently.

computer-use-privacy-boundary.test.ts is not new to #1888; it has asserted this
line since #1581, and #1888 added an elementId case to it. Both PRs are
therefore changing the same invariant, and this one changes it in place rather
than around it, so what crosses is visible in the diff and is spelled out in
the file, the commit message and the PR body.

Two things now cross. element_id, which is an index the model chose into an
observation it had already read, admitted only when it is a stable identifier
so an accessibility label arriving under that key is dropped — this is exactly
what #1888 wanted and it is subsumed here, in the tool's own casing.
coordinate, start_coordinate and region, which are geometry the model wrote
into the call itself, integers only so a mistyped value still degrades to a
shape.

The argument for carrying coordinates is that they are the model's own output,
already present in the assistant turn that produced them, and that withholding
them leaves a model that clicked a point and missed unable to see that it had
already tried that point. The argument against, that for set_value and type
these are what the user was having typed on their behalf, does not apply to
coordinates: it applies to text and value, and those still do not cross. They
keep their key and lose their value to , because the key has to stay or
the model reads its own history as a call it never made. text for press_key,
key, hold_key and secondary_action is a name from a closed set the executor
publishes, so it is carried; that is one argument name meaning six things, and
only two of the six come from outside the model.

element_identity is added to the host-only set. The Computer Use tool's own
permissionArgs resolves the model's element_id against the live observation and
attaches its identity, and permissionArgs is what this projection now runs on,
so without that the model would read back a key it cannot send whose value came
off the accessibility tree.

For #1888 specifically: when the two are merged, computer-action-label.ts
should read window_id and element_id rather than windowId and elementId, since
the persisted args are now in the tool's dialect. Nothing on main reads the
persisted Computer Use args in the approval-summary dialect — the permission
prompt reads permissionArgs and the approval scope key is computed from the raw
call — so that renderer is the only reader affected.

Non-blocking finding: the title and body are rewritten. 0fcfb46 removed the
capability declarations, so the branch no longer does what it was named for;
what remains is the model-facing projection and the action-list parity guard,
and the description now says so and says what was dropped and why.

Rebased onto current main, cleanly. @maka/core and @maka/runtime suites pass,
npx biome check is clean on the touched files, and node scripts/check-console.mjs
passes.

…ence it returns
Types only. Nothing selects them, nothing behaves differently, and every
existing test passes unchanged — this is the vocabulary a second executor
backend needs before it can be written against the same interface.
What it adds:
- `launchApp` on `CuDispatchBackend`. Starting an application is something an
executor can do and the host could not ask for, so a model that needed a
closed app had no move but to tell the user to open it.
- `truncated` on `CuObservation` and `actions` on `CuObservedElement`. A tree
that was cut short and an element that offers more than a press are both
facts the executor knows and the host had no field to carry, so they were
read, validated and then dropped.
- `COMPUTER_USE_SEMANTIC_ACTIONS`, so the set of element actions has one
written-down list rather than being spelled again at each place that
switches on it.
Optional throughout: a backend that provides none of it satisfies the
interface exactly as before.
… accepts
The previous version of this branch said nothing behaved differently. Four
things did.
`COMPUTER_USE_SEMANTIC_ACTIONS` declared `launch_app`, `scroll_element`,
`element_sequence` and `window_action`. None of those is in the tool's wire
enum on main; they exist only on the branch that adds the executor. The
constant feeds `APPROVAL_ACTIONS`, so `computerUseApprovalSummary` flipped
`knownAction` to true for four names the SDK rejects before the tool ever
runs, and `rememberForTurnAllowed` could evaluate true for two of them. A
person would have read an approval for an action that did not happen. The
constant is now scoped to what this schema accepts. Declaring an executor's
vocabulary is worth doing on the branch that also ships the executor.
`dispatch_refused` had no producer, and adding it to `COMPUTER_USE_ERROR_CODES`
widened `isComputerUseErrorCode` — the filter that decides whether a raw string
from the driver process is passed through or replaced with `capture_failed`.
That is a live trust boundary, not a declaration. Removed until something
produces it. The unproduced fields on the observation and outcome types are
left as they are: they have no runtime effect, and declaring the shape the
executor returns is what this branch is for.
The drift guard the constant's rationale rests on did not exist. The comment
promised a file that was never written, and the guard on main only walks from
the schemas to the catalog, which is the direction that catches a missing name
— not the direction this branch broke. `computer-use-schema-parity.test.ts`
now compares the two in both directions, with a negative control that runs the
real comparator over a fixed catalog so it keeps proving something on the day
the real one is correct.
`ComputerUseSettings` declared `enabled: false` and was registered nowhere:
not in `AppSettings`, not in `UpdateAppSettingsInput`, not in the defaults,
merge or normalize passes that every peer settings module goes through. In the
shipped app Computer Use tools are assembled with no enable check at all, so
the type told a reader the capability is off by default while it is always on.
Dropped. Wiring a real switch is a change with a UI and a migration, and it
does not belong under a types commit.
The model-call projection could not be used. Only `COMPUTER_USE_SEMANTIC_ACTIONS`
was exported from the package entry, so `computerUseModelCallArgs` — the
headline fix — was unreachable from `@maka/runtime`, and the defective
projection it exists to replace stayed in place. It is exported now and wired
into `projectPiToolArgs`, which is what writes the persisted `tool_call`
message and the `tool_start` event: the record the model reads back as its own
call. It had been reading the host's approval summary, which renames
`window_id` to `windowId`, adds two fields the model never sent, and drops
every argument it does not need.
And the projection withheld the wrong half of `text`. It keyed on argument
name, but `text` is six arguments under one name: the key for `press_key`,
`key` and `hold_key`, the element action name for `secondary_action`, a
substring of the window for `select_text`, and typed content for `type`.
Excluding all six is right for two of them. For the rest the model was reading
back `press_key ... text: <text>` and could not see which key it had pressed —
the exact failure the comment cited as motivation. It now keys on the action.
An action the tool does not accept has no plain set, so it falls through to
shapes rather than to whichever set was used last.
…ations nothing produces
The projection withheld coordinates on the grounds that they are
screen-derived. They are not: a coordinate is the model's own output,
four digits it chose and sent. Reduced to `<point>`, a model that clicked
[412, 88] and missed could not tell whether it had already tried that
point — the repeated-call shape this projection exists to make visible,
reintroduced by the projection itself. `coordinate`, `start_coordinate`
and `region` now come back whole when they are integers, and still
degrade to a shape when they are not.
An action the schema does not carry was recorded as `unknown`, so a model
whose call was rejected for the name it sent read its own history as a
call it never made and could not connect the rejection to the cause.
Collapsing an unrecognised name is `computerUseApprovalSummary`'s job,
where `knownAction` decides what a person is asked to allow; this record
is the model's own arguments, and the action is one of them. It is
bounded and redacted like every other model-supplied string.
The rest of the branch declared fourteen fields, one interface, one
backend method and two union members that nothing in the repository sets
and nothing reads. The parity guard added here covers the action-name
catalog and nothing else, so those declarations had nothing holding them
honest — which is the same condition the guard's own header describes:
`window_action` sat in the strict union and not on the wire and was
unreachable from the day it shipped, and nothing failed. This branch was
putting `window_action` back in that state, along with `scroll_element`.
They come with the executor that produces them, where a guard can check
that they arrive.
What is left is what has a consumer: `COMPUTER_USE_SEMANTIC_ACTIONS` with
the two-way parity check that keeps it and the wire enum in step, and
`computerUseModelCallArgs` wired into the Pi backend's tool_call record.
The projection was wired into PiAgentBackend, which the desktop app never
instantiates: boot.ts registers only ai-sdk and fake, and the pi backend
is reachable only from harbor-cell.ts behind MAKA_PI_*, driving an
external CLI, and it does not build a ToolRuntime at all. tool-runtime.ts
had a zero-byte diff, so on the shipping path nothing changed.
Measured on the real ToolRuntime before this commit, for a model that
sent {action:'press_key', app:'TextEdit', window_id:41,
observation_id:'obs-7', element_id:'e12', text:'cmd+s'}, the persisted
tool_start args were {action, approvalClass, rememberForTurnAllowed, app,
windowId:41, observationId} — a key the tool rejects, two host fields the
model never sent, no element, and a press_key with no key. model-history
replays event.content.args, so that is what the model read back of its
own call, and it went on calling it that way.
The line moves; the projection does not. The permission prompt still
reads permissionArgs and the approval scope key is still computed from
the raw call, so only what is written down changes.
Two things now cross the privacy boundary that did not, and
computer-use-privacy-boundary.test.ts says so where it asserts them.
element_id: an index the model chose into an observation it had already
read. Admitted only when it is a stable identifier, so an accessibility
label arriving under that key is dropped rather than persisted. Without
it every element action in a turn reads the same.
coordinate, start_coordinate, region: geometry the model wrote into the
call itself, integers only, so a mistyped value still degrades to a
shape. A coordinate is not read off the screen; withholding it left a
model that clicked a point and missed unable to see that it had already
tried that point.
What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed — those keep their key
and lose their value to <text>. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so
it is carried; that is one argument name meaning six things, and only two
of the six come from outside the model.
element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the
model's element_id against the live observation and attaches it, and
permissionArgs is what this projection now runs on, so without that the
model would read back a key it cannot send whose value came off the
accessibility tree.
The new assertion is on the tool_start event, not on the projection
function. A test that calls computerUseModelCallArgs directly stays green
while the runtime writes the approval summary, which is how this survived
being fixed once already.
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 03fe3d9 to 71df4c1CompareAugust 3, 2026 11:48
@hqhq1025
hqhq1025 merged commit 80eb37d into apache:mainAug 3, 2026
10 checks passed
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit that referenced this pull request Aug 3, 2026
…1888)
* feat(ui): give driving the computer its own row, and say what it did
Computer Use's display name is a noun — "Maka Computer" — so a turn that
observed a window, clicked a button and observed again rendered three
identical rows. `computerActionLabel` derives a row label from the call's
own arguments instead: 观察「计算器」窗口, 点击元素 e7.
The label is derived, never declared. The model is not given an `intent`
field to write; every word it produces is owned by the runtime, and a free
text field would be one more place it can be wrong.
What the renderer can read is `computerUseApprovalSummary(...)`, which
`ToolRuntime.executeTool` substitutes for the wire call before anything is
persisted. `element_id` had to be carried on that projection for a row to
name its target at all, and this adds it — but a shape filter is not a
privacy boundary. `stableIdentifier` admits `[A-Za-z0-9._:-]{1,256}`, which
is also the shape of an API key, and arguments are not validated before the
projection runs, so a model that put a token under `element_id` wrote it
verbatim into the persisted `tool_call`, onto `tool_start` and into the row
— while the same string under `observation_id` came out `[redacted]`. It
now takes the same `redactSecrets` pass `observationId` and `app` do.
Also puts `'computer'` on the wire as a tool activity kind and has the
Session decoder read the one list rather than a second hand-written copy of
it, which had already fallen behind and rejected the new kind outright.
Note for clients behind the runtime-host Session protocol: the `tool_start`
frame carries no `args`, so such a client gets the kind but cannot derive
the label. The desktop receives full SessionEvents over IPC and is
unaffected.
* fix(ui): read the Computer Use row's arguments in the dialect they are now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. #1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after #1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hqhq1025
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' fix(cu): let a model read back the Computer Use call it actually made by hqhq1025 · Pull Request #1891 · apache/maka · GitHub
Skip to content

fix(cu): let a model read back the Computer Use call it actually made - #1891

Merged
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types
Aug 3, 2026
Merged

fix(cu): let a model read back the Computer Use call it actually made#1891
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types

Conversation

@hqhq1025

@hqhq1025hqhq1025 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The record a model reads back of its own Computer Use call was the host's
approval projection, not the call. model-history.ts replays
event.content.args, and those args came from computerUseApprovalSummary,
which exists to decide and display a permission: it renames window_id to
windowId, adds approvalClass and rememberForTurnAllowed, and drops every
argument it does not need.

Measured on the real ToolRuntime, a model that sent

{"action":"press_key","app":"TextEdit","window_id":41,
"observation_id":"obs-7","element_id":"e12","text":"cmd+s"}

read back

{"action":"press_key","approvalClass":"keyboard_mutation",
"rememberForTurnAllowed":true,"app":"TextEdit","windowId":41,
"observationId":"obs-7"}

— a key the tool rejects, two fields it never sent, no element, and a
press_key with no key. A model imitates the shape it is shown, so it went on
calling it that way; a real desktop run failed six of eleven calls on shapes
copied from its own history.

This adds computerUseModelCallArgs: the same privacy rule, in the names the
tool accepts, and wires it at tool-runtime.ts, which is the line every
shipping backend goes through. The permission prompt still reads
permissionArgs and the approval scope key is still computed from the raw
call, so only what is written down changes.

What now crosses the privacy boundary

computer-use-privacy-boundary.test.ts asserted this line before this PR, and
it is updated in place rather than around, so the change is visible in the
diff. Two things cross that did not:

  • element_id — an index the model chose into an observation it had already
    read (e12). Admitted only when it is a stable identifier, so an
    accessibility label arriving under that key is dropped rather than persisted.
    Without it, ten element actions in a turn are indistinguishable.
  • coordinate, start_coordinate, region — geometry the model wrote into
    the call itself. Integers only, so a mistyped value still degrades to a
    shape. A coordinate is not read off the screen; withholding it leaves a model
    that clicked a point and missed unable to see that it had already tried that
    point.

What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed. Those keep their key and
lose their value to <text> — the key has to stay, or the model reads its own
history as a call it never made. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so it is
carried: one argument name meaning six things, and only two of the six come
from outside the model.

element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the model's
element_id against the live observation and attaches it, and permissionArgs
is what this projection runs on.

Also here

COMPUTER_USE_SEMANTIC_ACTIONS replaces a hand-written action list that had
already drifted: window_action was added to the strict union and not to the
list, so every window move, resize and minimise would have summarised as
unknown — in the approval a person reads, and in the record the model reads
back. computer-use-schema-parity.test.ts now compares the two in both
directions.

Relationship to #1888

#1888 reached the same conclusion from the renderer's side and carried
elementId in the approval summary so its labels could tell two element rows
apart. That intent is subsumed here: element_id is carried, in the tool's own
casing. When the two are merged, #1888's computer-action-label.ts should read
window_id and element_id rather than windowId and elementId, and the
elementId case it added to computer-use-privacy-boundary.test.ts is covered
by the projection assertion in this branch.

Not here any more

The executor capability declarations this PR was originally titled for were
removed in 0fcfb46: nothing produced them, so they were a type surface with
no writer. The title and this description are rewritten to match what the
branch actually does.

Verification

The new assertion is on the tool_start event, not on the projection function.
Reverted to computerUseApprovalSummary at tool-runtime.ts and rebuilt, it
fails with exactly the diff quoted above; restored, it passes. A test that
calls computerUseModelCallArgs directly stays green either way, which is how
this survived being fixed once already in PiAgentBackend — a backend the
desktop app never instantiates.

@maka/core and @maka/runtime suites pass; npx biome check clean on the
touched files; node scripts/check-console.mjs passes. Rebased onto current
main, cleanly.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all six findings. The claim in the PR body that nothing behaves differently was wrong, and finding 1 was the reason.

Finding 1, resolution: I scoped the constant to what main's schema accepts rather than stacking the PR on the executor branch. Verified first — main's wire enum at computer-use-tools.ts is list_apps, observe, click_element, set_value, select_text, secondary_action, press_key plus CU_ACTION_TYPES, and none of launch_app, scroll_element, element_sequence or window_action is in it. The four names went into APPROVAL_ACTIONS through COMPUTER_USE_SEMANTIC_ACTIONS, so computerUseApprovalSummary flipped knownAction to true for calls the SDK rejects before the tool runs, and rememberForTurnAllowed could evaluate true for the two that also joined SEMANTIC_ACTIONS. I chose scoping over stacking for two reasons. The constant's stated purpose is to be held against the tool schema, so declaring names the schema does not have is the same drift it exists to prevent, only in the direction nothing was checking. And stacking would make the PR unmergeable until the executor lands, which delays the parts that are correct today. The executor's vocabulary belongs on the branch that also ships the executor.

Finding 2: the promised file did not exist, but a guard did — computer-use-schema-parity.test.ts, added in #1870 after this branch was cut. It only walks from the schemas to the catalog, which catches a missing name and not an extra one, so the widening in finding 1 would have passed it. I added the reverse comparison to that file rather than creating the second one the comment named, and corrected the comment to point at the real file. Negative control: reintroducing the four names turns the new check red and it names all four. There is also an in-file control that runs the same comparator over a fixed catalog, so it keeps proving the comparator can return something on the day the real catalog is correct.

Finding 3: confirmed. Only COMPUTER_USE_SEMANTIC_ACTIONS was exported, so computerUseModelCallArgs was unreachable from @maka/runtime and the whole headline fix was dead code while the harmful widening was the only part that shipped. The five missing symbols are exported now, and the projection is wired into projectPiToolArgs in pi-agent-backend.ts, which writes the persisted tool_call message and the tool_start event. Negative control: reverting that one line turns the pi-agent-backend projection test red with the approval dialect back in the diff.

Finding 4: confirmed and dropped rather than wired. ComputerUseSettings appears in none of AppSettings, UpdateAppSettingsInput, the defaults, the merge or the normalize pass, unlike every peer settings module, and select-backend.ts assembles the CU tools with no enable check anywhere. A type that says enabled: false while the shipped app always has it on is worse than no type. Wiring a real switch needs a UI and a migration and does not belong under a types commit.

Finding 5: confirmed, and it is exactly as described. text is six arguments under one name — the key for press_key, key and hold_key, the element action name for secondary_action, a screen substring for select_text, typed content for type. MODEL_CALL_PLAIN_VALUES is now a map from action to the argument names that are plain for that action. An action the tool does not accept has no entry, so it falls through to shapes rather than to whichever set was used last. Negative control: reverting to the name-keyed set turns two tests red, showing press_key coming back as text: .

Finding 6, partially rejected with evidence. dispatch_refused is gone. It is not a declaration — COMPUTER_USE_ERROR_CODES is a runtime value feeding isComputerUseErrorCode, which decides at cua-driver-result.ts:88 whether a raw string from the driver process is trusted or replaced with capture_failed. Widening that with no producer is a live change. The other five named items are different in kind. messageIsAppTextFree, truncated, actions, subrole and placeholder are optional fields on TypeScript interfaces with no runtime effect, and they are not distinguishable from about ten other unproduced declarations in the same diff that the review accepted — focused, appAlias, obscuringRects, menu, query, CuLaunchedApp, scroll_element, window_action, launchApp. Dropping the named five and keeping those would be arbitrary; dropping all of them empties the branch of its purpose. Declaring the shape an executor returns is what this branch is for, so they stay, and the absent-means-withheld polarity on messageIsAppTextFree is kept as suggested.

Also rebased onto current main, which the branch was one commit behind — the missing commit was #1870, the parity test finding 2 turns out to depend on.

Test delta, measured against a baseline taken on clean origin/main on the same machine. This machine's Node has no node:sqlite, so @maka/runtime carries 64 pre-existing failing test names there. After the change: 2722 tests, zero new failures and zero fixed against that set. @maka/core goes 703/703 to 710/710, the seven new ones being the model-call projection tests. biome check clean on the six touched files, check-console passes.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Second-round review addressed. Two findings fixed, one dropped after checking, one scope decision.

Coordinates are back. The doc said typed text, written values and coordinates are screen-derived and stay out, and that is false for the third one: a coordinate is the model's own output, four digits it chose and sent. Reduced to <point>, a model that clicked [412, 88] and missed could not tell whether it had already tried that point, which is the repeated-call shape this projection exists to make visible. coordinate, start_coordinate and region now come back whole when they are integers and still degrade to a shape when they are not, so a malformed argument does not become a leak channel.

The out-of-enum action is reported as sent. On the AI SDK path the question is moot — a schema-rejected call never produces a tool-call part, so settleToolCall is never reached and no tool_start is emitted. But this projection's only consumer is the Pi backend, where tool_start arrives as a transport frame that Maka does not validate, so the case is live there. It is also wrong in principle: this record is the model's own arguments, and the action is one of them. Collapsing an unrecognised name is computerUseApprovalSummary's job, where knownAction decides what a person is asked to allow; a test now pins that the two functions disagree on purpose. The action is bounded and redacted like every other model-supplied string.

The declared-and-unconsumed set is dropped. launchApp, CuLaunchedApp, truncated, actions, subrole, placeholder, focused, menu, query, appAlias, obscuringRects, targetWindowId, targetStacking, messageIsAppTextFree, and the scroll_element and window_action members of CuSemanticAction — zero producers and zero consumers, confirmed by grep across packages and apps. The reason is the parity guard's own header: window_action sat in the strict union and not on the wire, was unreachable from the day it shipped, and nothing failed, because a declaration with no producer has nothing to check it. This branch was putting window_action back into that state with scroll_element beside it. computer-use-types.ts is now byte-identical to main; the additions come with the executor that produces them, where a guard can assert they arrive. The PR body has been rewritten so it no longer claims them.

What is left is what has a consumer: COMPUTER_USE_SEMANTIC_ACTIONS with the two-way parity check, and computerUseModelCallArgs wired into the Pi backend.

Negative controls, run rather than asserted. Reverting the geometry branch turned the coordinate tests red on coordinate: '<point>' against [412, 88]; restoring the APPROVAL_ACTIONS gate on action turned the third red on unknown against element_sequence, and the Pi backend's own projection test with it. Both back to green after restoring.

One thing I did not change, because it is outside this branch: tool-runtime.ts:776 still writes computerUseApprovalSummary(permissionArgs) into both the tool_start event and the persisted tool_call message on the AI SDK path. That is the same defect this PR fixes, on the main backend rather than the Pi one. It needs its own change, since the same value also feeds describeToolIntent, the durable args hash and the approval record.

Test delta against a clean rebuild of main on this machine: none in packages/core or the Computer Use tests. The rest of the run is shell, filesystem and host-process timing under load 40 on this box; each one passes when its file is run alone.

@hqhq1025hqhq1025 changed the title feat(cu): declare the capabilities an executor can have, and the evidence it returnsfix(cu): let a model read back the Computer Use call it actually madeAug 3, 2026
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 0fcfb46 to 03fe3d9CompareAugust 3, 2026 11:11
@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Blocking finding addressed: the fix is moved to the path the shipping app runs.

It was wired into PiAgentBackend, which the desktop app never instantiates.
boot.ts registers only ai-sdk and fake; the pi backend is reachable only from
harbor-cell.ts behind MAKA_PI_*, driving an external CLI, and it does not
construct a ToolRuntime at all. tool-runtime.ts had a zero-byte diff, so on the
shipping path nothing changed. Measured on the real ToolRuntime, a model that
sent action press_key with app, window_id 41, observation_id obs-7, element_id
e12 and text cmd+s read back action, approvalClass, rememberForTurnAllowed,
app, windowId 41, observationId — a key the tool rejects, two fields it never
sent, no element, and a press_key with no key. That is now the one line the PR
changes in production.

The new assertion is on the tool_start event, which is what model-history.ts
replays, rather than on the projection function. Reverted to
computerUseApprovalSummary and rebuilt, it fails with exactly the diff above;
restored, it passes. A test calling computerUseModelCallArgs directly stays
green either way, which is how this survived being fixed once already.

On the coupling with #1888 and the privacy boundary, since the description
asked that it not be resolved silently.

computer-use-privacy-boundary.test.ts is not new to #1888; it has asserted this
line since #1581, and #1888 added an elementId case to it. Both PRs are
therefore changing the same invariant, and this one changes it in place rather
than around it, so what crosses is visible in the diff and is spelled out in
the file, the commit message and the PR body.

Two things now cross. element_id, which is an index the model chose into an
observation it had already read, admitted only when it is a stable identifier
so an accessibility label arriving under that key is dropped — this is exactly
what #1888 wanted and it is subsumed here, in the tool's own casing.
coordinate, start_coordinate and region, which are geometry the model wrote
into the call itself, integers only so a mistyped value still degrades to a
shape.

The argument for carrying coordinates is that they are the model's own output,
already present in the assistant turn that produced them, and that withholding
them leaves a model that clicked a point and missed unable to see that it had
already tried that point. The argument against, that for set_value and type
these are what the user was having typed on their behalf, does not apply to
coordinates: it applies to text and value, and those still do not cross. They
keep their key and lose their value to , because the key has to stay or
the model reads its own history as a call it never made. text for press_key,
key, hold_key and secondary_action is a name from a closed set the executor
publishes, so it is carried; that is one argument name meaning six things, and
only two of the six come from outside the model.

element_identity is added to the host-only set. The Computer Use tool's own
permissionArgs resolves the model's element_id against the live observation and
attaches its identity, and permissionArgs is what this projection now runs on,
so without that the model would read back a key it cannot send whose value came
off the accessibility tree.

For #1888 specifically: when the two are merged, computer-action-label.ts
should read window_id and element_id rather than windowId and elementId, since
the persisted args are now in the tool's dialect. Nothing on main reads the
persisted Computer Use args in the approval-summary dialect — the permission
prompt reads permissionArgs and the approval scope key is computed from the raw
call — so that renderer is the only reader affected.

Non-blocking finding: the title and body are rewritten. 0fcfb46 removed the
capability declarations, so the branch no longer does what it was named for;
what remains is the model-facing projection and the action-list parity guard,
and the description now says so and says what was dropped and why.

Rebased onto current main, cleanly. @maka/core and @maka/runtime suites pass,
npx biome check is clean on the touched files, and node scripts/check-console.mjs
passes.

…ence it returns
Types only. Nothing selects them, nothing behaves differently, and every
existing test passes unchanged — this is the vocabulary a second executor
backend needs before it can be written against the same interface.
What it adds:
- `launchApp` on `CuDispatchBackend`. Starting an application is something an
executor can do and the host could not ask for, so a model that needed a
closed app had no move but to tell the user to open it.
- `truncated` on `CuObservation` and `actions` on `CuObservedElement`. A tree
that was cut short and an element that offers more than a press are both
facts the executor knows and the host had no field to carry, so they were
read, validated and then dropped.
- `COMPUTER_USE_SEMANTIC_ACTIONS`, so the set of element actions has one
written-down list rather than being spelled again at each place that
switches on it.
Optional throughout: a backend that provides none of it satisfies the
interface exactly as before.
… accepts
The previous version of this branch said nothing behaved differently. Four
things did.
`COMPUTER_USE_SEMANTIC_ACTIONS` declared `launch_app`, `scroll_element`,
`element_sequence` and `window_action`. None of those is in the tool's wire
enum on main; they exist only on the branch that adds the executor. The
constant feeds `APPROVAL_ACTIONS`, so `computerUseApprovalSummary` flipped
`knownAction` to true for four names the SDK rejects before the tool ever
runs, and `rememberForTurnAllowed` could evaluate true for two of them. A
person would have read an approval for an action that did not happen. The
constant is now scoped to what this schema accepts. Declaring an executor's
vocabulary is worth doing on the branch that also ships the executor.
`dispatch_refused` had no producer, and adding it to `COMPUTER_USE_ERROR_CODES`
widened `isComputerUseErrorCode` — the filter that decides whether a raw string
from the driver process is passed through or replaced with `capture_failed`.
That is a live trust boundary, not a declaration. Removed until something
produces it. The unproduced fields on the observation and outcome types are
left as they are: they have no runtime effect, and declaring the shape the
executor returns is what this branch is for.
The drift guard the constant's rationale rests on did not exist. The comment
promised a file that was never written, and the guard on main only walks from
the schemas to the catalog, which is the direction that catches a missing name
— not the direction this branch broke. `computer-use-schema-parity.test.ts`
now compares the two in both directions, with a negative control that runs the
real comparator over a fixed catalog so it keeps proving something on the day
the real one is correct.
`ComputerUseSettings` declared `enabled: false` and was registered nowhere:
not in `AppSettings`, not in `UpdateAppSettingsInput`, not in the defaults,
merge or normalize passes that every peer settings module goes through. In the
shipped app Computer Use tools are assembled with no enable check at all, so
the type told a reader the capability is off by default while it is always on.
Dropped. Wiring a real switch is a change with a UI and a migration, and it
does not belong under a types commit.
The model-call projection could not be used. Only `COMPUTER_USE_SEMANTIC_ACTIONS`
was exported from the package entry, so `computerUseModelCallArgs` — the
headline fix — was unreachable from `@maka/runtime`, and the defective
projection it exists to replace stayed in place. It is exported now and wired
into `projectPiToolArgs`, which is what writes the persisted `tool_call`
message and the `tool_start` event: the record the model reads back as its own
call. It had been reading the host's approval summary, which renames
`window_id` to `windowId`, adds two fields the model never sent, and drops
every argument it does not need.
And the projection withheld the wrong half of `text`. It keyed on argument
name, but `text` is six arguments under one name: the key for `press_key`,
`key` and `hold_key`, the element action name for `secondary_action`, a
substring of the window for `select_text`, and typed content for `type`.
Excluding all six is right for two of them. For the rest the model was reading
back `press_key ... text: <text>` and could not see which key it had pressed —
the exact failure the comment cited as motivation. It now keys on the action.
An action the tool does not accept has no plain set, so it falls through to
shapes rather than to whichever set was used last.
…ations nothing produces
The projection withheld coordinates on the grounds that they are
screen-derived. They are not: a coordinate is the model's own output,
four digits it chose and sent. Reduced to `<point>`, a model that clicked
[412, 88] and missed could not tell whether it had already tried that
point — the repeated-call shape this projection exists to make visible,
reintroduced by the projection itself. `coordinate`, `start_coordinate`
and `region` now come back whole when they are integers, and still
degrade to a shape when they are not.
An action the schema does not carry was recorded as `unknown`, so a model
whose call was rejected for the name it sent read its own history as a
call it never made and could not connect the rejection to the cause.
Collapsing an unrecognised name is `computerUseApprovalSummary`'s job,
where `knownAction` decides what a person is asked to allow; this record
is the model's own arguments, and the action is one of them. It is
bounded and redacted like every other model-supplied string.
The rest of the branch declared fourteen fields, one interface, one
backend method and two union members that nothing in the repository sets
and nothing reads. The parity guard added here covers the action-name
catalog and nothing else, so those declarations had nothing holding them
honest — which is the same condition the guard's own header describes:
`window_action` sat in the strict union and not on the wire and was
unreachable from the day it shipped, and nothing failed. This branch was
putting `window_action` back in that state, along with `scroll_element`.
They come with the executor that produces them, where a guard can check
that they arrive.
What is left is what has a consumer: `COMPUTER_USE_SEMANTIC_ACTIONS` with
the two-way parity check that keeps it and the wire enum in step, and
`computerUseModelCallArgs` wired into the Pi backend's tool_call record.
The projection was wired into PiAgentBackend, which the desktop app never
instantiates: boot.ts registers only ai-sdk and fake, and the pi backend
is reachable only from harbor-cell.ts behind MAKA_PI_*, driving an
external CLI, and it does not build a ToolRuntime at all. tool-runtime.ts
had a zero-byte diff, so on the shipping path nothing changed.
Measured on the real ToolRuntime before this commit, for a model that
sent {action:'press_key', app:'TextEdit', window_id:41,
observation_id:'obs-7', element_id:'e12', text:'cmd+s'}, the persisted
tool_start args were {action, approvalClass, rememberForTurnAllowed, app,
windowId:41, observationId} — a key the tool rejects, two host fields the
model never sent, no element, and a press_key with no key. model-history
replays event.content.args, so that is what the model read back of its
own call, and it went on calling it that way.
The line moves; the projection does not. The permission prompt still
reads permissionArgs and the approval scope key is still computed from
the raw call, so only what is written down changes.
Two things now cross the privacy boundary that did not, and
computer-use-privacy-boundary.test.ts says so where it asserts them.
element_id: an index the model chose into an observation it had already
read. Admitted only when it is a stable identifier, so an accessibility
label arriving under that key is dropped rather than persisted. Without
it every element action in a turn reads the same.
coordinate, start_coordinate, region: geometry the model wrote into the
call itself, integers only, so a mistyped value still degrades to a
shape. A coordinate is not read off the screen; withholding it left a
model that clicked a point and missed unable to see that it had already
tried that point.
What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed — those keep their key
and lose their value to <text>. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so
it is carried; that is one argument name meaning six things, and only two
of the six come from outside the model.
element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the
model's element_id against the live observation and attaches it, and
permissionArgs is what this projection now runs on, so without that the
model would read back a key it cannot send whose value came off the
accessibility tree.
The new assertion is on the tool_start event, not on the projection
function. A test that calls computerUseModelCallArgs directly stays green
while the runtime writes the approval summary, which is how this survived
being fixed once already.
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 03fe3d9 to 71df4c1CompareAugust 3, 2026 11:48
@hqhq1025
hqhq1025 merged commit 80eb37d into apache:mainAug 3, 2026
10 checks passed
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit that referenced this pull request Aug 3, 2026
…1888)
* feat(ui): give driving the computer its own row, and say what it did
Computer Use's display name is a noun — "Maka Computer" — so a turn that
observed a window, clicked a button and observed again rendered three
identical rows. `computerActionLabel` derives a row label from the call's
own arguments instead: 观察「计算器」窗口, 点击元素 e7.
The label is derived, never declared. The model is not given an `intent`
field to write; every word it produces is owned by the runtime, and a free
text field would be one more place it can be wrong.
What the renderer can read is `computerUseApprovalSummary(...)`, which
`ToolRuntime.executeTool` substitutes for the wire call before anything is
persisted. `element_id` had to be carried on that projection for a row to
name its target at all, and this adds it — but a shape filter is not a
privacy boundary. `stableIdentifier` admits `[A-Za-z0-9._:-]{1,256}`, which
is also the shape of an API key, and arguments are not validated before the
projection runs, so a model that put a token under `element_id` wrote it
verbatim into the persisted `tool_call`, onto `tool_start` and into the row
— while the same string under `observation_id` came out `[redacted]`. It
now takes the same `redactSecrets` pass `observationId` and `app` do.
Also puts `'computer'` on the wire as a tool activity kind and has the
Session decoder read the one list rather than a second hand-written copy of
it, which had already fallen behind and rejected the new kind outright.
Note for clients behind the runtime-host Session protocol: the `tool_start`
frame carries no `args`, so such a client gets the kind but cannot derive
the label. The desktop receives full SessionEvents over IPC and is
unaffected.
* fix(ui): read the Computer Use row's arguments in the dialect they are now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. #1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after #1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hqhq1025
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); fix(cu): let a model read back the Computer Use call it actually made by hqhq1025 · Pull Request #1891 · apache/maka · GitHub
Skip to content

fix(cu): let a model read back the Computer Use call it actually made - #1891

Merged
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types
Aug 3, 2026
Merged

fix(cu): let a model read back the Computer Use call it actually made#1891
hqhq1025 merged 4 commits into
apache:mainfrom
hqhq1025:pr/cu-capability-types

Conversation

@hqhq1025

@hqhq1025hqhq1025 commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

The record a model reads back of its own Computer Use call was the host's
approval projection, not the call. model-history.ts replays
event.content.args, and those args came from computerUseApprovalSummary,
which exists to decide and display a permission: it renames window_id to
windowId, adds approvalClass and rememberForTurnAllowed, and drops every
argument it does not need.

Measured on the real ToolRuntime, a model that sent

{"action":"press_key","app":"TextEdit","window_id":41,
"observation_id":"obs-7","element_id":"e12","text":"cmd+s"}

read back

{"action":"press_key","approvalClass":"keyboard_mutation",
"rememberForTurnAllowed":true,"app":"TextEdit","windowId":41,
"observationId":"obs-7"}

— a key the tool rejects, two fields it never sent, no element, and a
press_key with no key. A model imitates the shape it is shown, so it went on
calling it that way; a real desktop run failed six of eleven calls on shapes
copied from its own history.

This adds computerUseModelCallArgs: the same privacy rule, in the names the
tool accepts, and wires it at tool-runtime.ts, which is the line every
shipping backend goes through. The permission prompt still reads
permissionArgs and the approval scope key is still computed from the raw
call, so only what is written down changes.

What now crosses the privacy boundary

computer-use-privacy-boundary.test.ts asserted this line before this PR, and
it is updated in place rather than around, so the change is visible in the
diff. Two things cross that did not:

  • element_id — an index the model chose into an observation it had already
    read (e12). Admitted only when it is a stable identifier, so an
    accessibility label arriving under that key is dropped rather than persisted.
    Without it, ten element actions in a turn are indistinguishable.
  • coordinate, start_coordinate, region — geometry the model wrote into
    the call itself. Integers only, so a mistyped value still degrades to a
    shape. A coordinate is not read off the screen; withholding it leaves a model
    that clicked a point and missed unable to see that it had already tried that
    point.

What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed. Those keep their key and
lose their value to <text> — the key has to stay, or the model reads its own
history as a call it never made. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so it is
carried: one argument name meaning six things, and only two of the six come
from outside the model.

element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the model's
element_id against the live observation and attaches it, and permissionArgs
is what this projection runs on.

Also here

COMPUTER_USE_SEMANTIC_ACTIONS replaces a hand-written action list that had
already drifted: window_action was added to the strict union and not to the
list, so every window move, resize and minimise would have summarised as
unknown — in the approval a person reads, and in the record the model reads
back. computer-use-schema-parity.test.ts now compares the two in both
directions.

Relationship to #1888

#1888 reached the same conclusion from the renderer's side and carried
elementId in the approval summary so its labels could tell two element rows
apart. That intent is subsumed here: element_id is carried, in the tool's own
casing. When the two are merged, #1888's computer-action-label.ts should read
window_id and element_id rather than windowId and elementId, and the
elementId case it added to computer-use-privacy-boundary.test.ts is covered
by the projection assertion in this branch.

Not here any more

The executor capability declarations this PR was originally titled for were
removed in 0fcfb46: nothing produced them, so they were a type surface with
no writer. The title and this description are rewritten to match what the
branch actually does.

Verification

The new assertion is on the tool_start event, not on the projection function.
Reverted to computerUseApprovalSummary at tool-runtime.ts and rebuilt, it
fails with exactly the diff quoted above; restored, it passes. A test that
calls computerUseModelCallArgs directly stays green either way, which is how
this survived being fixed once already in PiAgentBackend — a backend the
desktop app never instantiates.

@maka/core and @maka/runtime suites pass; npx biome check clean on the
touched files; node scripts/check-console.mjs passes. Rebased onto current
main, cleanly.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Pushed fixes for all six findings. The claim in the PR body that nothing behaves differently was wrong, and finding 1 was the reason.

Finding 1, resolution: I scoped the constant to what main's schema accepts rather than stacking the PR on the executor branch. Verified first — main's wire enum at computer-use-tools.ts is list_apps, observe, click_element, set_value, select_text, secondary_action, press_key plus CU_ACTION_TYPES, and none of launch_app, scroll_element, element_sequence or window_action is in it. The four names went into APPROVAL_ACTIONS through COMPUTER_USE_SEMANTIC_ACTIONS, so computerUseApprovalSummary flipped knownAction to true for calls the SDK rejects before the tool runs, and rememberForTurnAllowed could evaluate true for the two that also joined SEMANTIC_ACTIONS. I chose scoping over stacking for two reasons. The constant's stated purpose is to be held against the tool schema, so declaring names the schema does not have is the same drift it exists to prevent, only in the direction nothing was checking. And stacking would make the PR unmergeable until the executor lands, which delays the parts that are correct today. The executor's vocabulary belongs on the branch that also ships the executor.

Finding 2: the promised file did not exist, but a guard did — computer-use-schema-parity.test.ts, added in #1870 after this branch was cut. It only walks from the schemas to the catalog, which catches a missing name and not an extra one, so the widening in finding 1 would have passed it. I added the reverse comparison to that file rather than creating the second one the comment named, and corrected the comment to point at the real file. Negative control: reintroducing the four names turns the new check red and it names all four. There is also an in-file control that runs the same comparator over a fixed catalog, so it keeps proving the comparator can return something on the day the real catalog is correct.

Finding 3: confirmed. Only COMPUTER_USE_SEMANTIC_ACTIONS was exported, so computerUseModelCallArgs was unreachable from @maka/runtime and the whole headline fix was dead code while the harmful widening was the only part that shipped. The five missing symbols are exported now, and the projection is wired into projectPiToolArgs in pi-agent-backend.ts, which writes the persisted tool_call message and the tool_start event. Negative control: reverting that one line turns the pi-agent-backend projection test red with the approval dialect back in the diff.

Finding 4: confirmed and dropped rather than wired. ComputerUseSettings appears in none of AppSettings, UpdateAppSettingsInput, the defaults, the merge or the normalize pass, unlike every peer settings module, and select-backend.ts assembles the CU tools with no enable check anywhere. A type that says enabled: false while the shipped app always has it on is worse than no type. Wiring a real switch needs a UI and a migration and does not belong under a types commit.

Finding 5: confirmed, and it is exactly as described. text is six arguments under one name — the key for press_key, key and hold_key, the element action name for secondary_action, a screen substring for select_text, typed content for type. MODEL_CALL_PLAIN_VALUES is now a map from action to the argument names that are plain for that action. An action the tool does not accept has no entry, so it falls through to shapes rather than to whichever set was used last. Negative control: reverting to the name-keyed set turns two tests red, showing press_key coming back as text: .

Finding 6, partially rejected with evidence. dispatch_refused is gone. It is not a declaration — COMPUTER_USE_ERROR_CODES is a runtime value feeding isComputerUseErrorCode, which decides at cua-driver-result.ts:88 whether a raw string from the driver process is trusted or replaced with capture_failed. Widening that with no producer is a live change. The other five named items are different in kind. messageIsAppTextFree, truncated, actions, subrole and placeholder are optional fields on TypeScript interfaces with no runtime effect, and they are not distinguishable from about ten other unproduced declarations in the same diff that the review accepted — focused, appAlias, obscuringRects, menu, query, CuLaunchedApp, scroll_element, window_action, launchApp. Dropping the named five and keeping those would be arbitrary; dropping all of them empties the branch of its purpose. Declaring the shape an executor returns is what this branch is for, so they stay, and the absent-means-withheld polarity on messageIsAppTextFree is kept as suggested.

Also rebased onto current main, which the branch was one commit behind — the missing commit was #1870, the parity test finding 2 turns out to depend on.

Test delta, measured against a baseline taken on clean origin/main on the same machine. This machine's Node has no node:sqlite, so @maka/runtime carries 64 pre-existing failing test names there. After the change: 2722 tests, zero new failures and zero fixed against that set. @maka/core goes 703/703 to 710/710, the seven new ones being the model-call projection tests. biome check clean on the six touched files, check-console passes.

@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Second-round review addressed. Two findings fixed, one dropped after checking, one scope decision.

Coordinates are back. The doc said typed text, written values and coordinates are screen-derived and stay out, and that is false for the third one: a coordinate is the model's own output, four digits it chose and sent. Reduced to <point>, a model that clicked [412, 88] and missed could not tell whether it had already tried that point, which is the repeated-call shape this projection exists to make visible. coordinate, start_coordinate and region now come back whole when they are integers and still degrade to a shape when they are not, so a malformed argument does not become a leak channel.

The out-of-enum action is reported as sent. On the AI SDK path the question is moot — a schema-rejected call never produces a tool-call part, so settleToolCall is never reached and no tool_start is emitted. But this projection's only consumer is the Pi backend, where tool_start arrives as a transport frame that Maka does not validate, so the case is live there. It is also wrong in principle: this record is the model's own arguments, and the action is one of them. Collapsing an unrecognised name is computerUseApprovalSummary's job, where knownAction decides what a person is asked to allow; a test now pins that the two functions disagree on purpose. The action is bounded and redacted like every other model-supplied string.

The declared-and-unconsumed set is dropped. launchApp, CuLaunchedApp, truncated, actions, subrole, placeholder, focused, menu, query, appAlias, obscuringRects, targetWindowId, targetStacking, messageIsAppTextFree, and the scroll_element and window_action members of CuSemanticAction — zero producers and zero consumers, confirmed by grep across packages and apps. The reason is the parity guard's own header: window_action sat in the strict union and not on the wire, was unreachable from the day it shipped, and nothing failed, because a declaration with no producer has nothing to check it. This branch was putting window_action back into that state with scroll_element beside it. computer-use-types.ts is now byte-identical to main; the additions come with the executor that produces them, where a guard can assert they arrive. The PR body has been rewritten so it no longer claims them.

What is left is what has a consumer: COMPUTER_USE_SEMANTIC_ACTIONS with the two-way parity check, and computerUseModelCallArgs wired into the Pi backend.

Negative controls, run rather than asserted. Reverting the geometry branch turned the coordinate tests red on coordinate: '<point>' against [412, 88]; restoring the APPROVAL_ACTIONS gate on action turned the third red on unknown against element_sequence, and the Pi backend's own projection test with it. Both back to green after restoring.

One thing I did not change, because it is outside this branch: tool-runtime.ts:776 still writes computerUseApprovalSummary(permissionArgs) into both the tool_start event and the persisted tool_call message on the AI SDK path. That is the same defect this PR fixes, on the main backend rather than the Pi one. It needs its own change, since the same value also feeds describeToolIntent, the durable args hash and the approval record.

Test delta against a clean rebuild of main on this machine: none in packages/core or the Computer Use tests. The rest of the run is shell, filesystem and host-process timing under load 40 on this box; each one passes when its file is run alone.

@hqhq1025hqhq1025 changed the title feat(cu): declare the capabilities an executor can have, and the evidence it returnsfix(cu): let a model read back the Computer Use call it actually madeAug 3, 2026
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 0fcfb46 to 03fe3d9CompareAugust 3, 2026 11:11
@hqhq1025

Copy link
Copy Markdown
ContributorAuthor

Blocking finding addressed: the fix is moved to the path the shipping app runs.

It was wired into PiAgentBackend, which the desktop app never instantiates.
boot.ts registers only ai-sdk and fake; the pi backend is reachable only from
harbor-cell.ts behind MAKA_PI_*, driving an external CLI, and it does not
construct a ToolRuntime at all. tool-runtime.ts had a zero-byte diff, so on the
shipping path nothing changed. Measured on the real ToolRuntime, a model that
sent action press_key with app, window_id 41, observation_id obs-7, element_id
e12 and text cmd+s read back action, approvalClass, rememberForTurnAllowed,
app, windowId 41, observationId — a key the tool rejects, two fields it never
sent, no element, and a press_key with no key. That is now the one line the PR
changes in production.

The new assertion is on the tool_start event, which is what model-history.ts
replays, rather than on the projection function. Reverted to
computerUseApprovalSummary and rebuilt, it fails with exactly the diff above;
restored, it passes. A test calling computerUseModelCallArgs directly stays
green either way, which is how this survived being fixed once already.

On the coupling with #1888 and the privacy boundary, since the description
asked that it not be resolved silently.

computer-use-privacy-boundary.test.ts is not new to #1888; it has asserted this
line since #1581, and #1888 added an elementId case to it. Both PRs are
therefore changing the same invariant, and this one changes it in place rather
than around it, so what crosses is visible in the diff and is spelled out in
the file, the commit message and the PR body.

Two things now cross. element_id, which is an index the model chose into an
observation it had already read, admitted only when it is a stable identifier
so an accessibility label arriving under that key is dropped — this is exactly
what #1888 wanted and it is subsumed here, in the tool's own casing.
coordinate, start_coordinate and region, which are geometry the model wrote
into the call itself, integers only so a mistyped value still degrades to a
shape.

The argument for carrying coordinates is that they are the model's own output,
already present in the assistant turn that produced them, and that withholding
them leaves a model that clicked a point and missed unable to see that it had
already tried that point. The argument against, that for set_value and type
these are what the user was having typed on their behalf, does not apply to
coordinates: it applies to text and value, and those still do not cross. They
keep their key and lose their value to , because the key has to stay or
the model reads its own history as a call it never made. text for press_key,
key, hold_key and secondary_action is a name from a closed set the executor
publishes, so it is carried; that is one argument name meaning six things, and
only two of the six come from outside the model.

element_identity is added to the host-only set. The Computer Use tool's own
permissionArgs resolves the model's element_id against the live observation and
attaches its identity, and permissionArgs is what this projection now runs on,
so without that the model would read back a key it cannot send whose value came
off the accessibility tree.

For #1888 specifically: when the two are merged, computer-action-label.ts
should read window_id and element_id rather than windowId and elementId, since
the persisted args are now in the tool's dialect. Nothing on main reads the
persisted Computer Use args in the approval-summary dialect — the permission
prompt reads permissionArgs and the approval scope key is computed from the raw
call — so that renderer is the only reader affected.

Non-blocking finding: the title and body are rewritten. 0fcfb46 removed the
capability declarations, so the branch no longer does what it was named for;
what remains is the model-facing projection and the action-list parity guard,
and the description now says so and says what was dropped and why.

Rebased onto current main, cleanly. @maka/core and @maka/runtime suites pass,
npx biome check is clean on the touched files, and node scripts/check-console.mjs
passes.

…ence it returns
Types only. Nothing selects them, nothing behaves differently, and every
existing test passes unchanged — this is the vocabulary a second executor
backend needs before it can be written against the same interface.
What it adds:
- `launchApp` on `CuDispatchBackend`. Starting an application is something an
executor can do and the host could not ask for, so a model that needed a
closed app had no move but to tell the user to open it.
- `truncated` on `CuObservation` and `actions` on `CuObservedElement`. A tree
that was cut short and an element that offers more than a press are both
facts the executor knows and the host had no field to carry, so they were
read, validated and then dropped.
- `COMPUTER_USE_SEMANTIC_ACTIONS`, so the set of element actions has one
written-down list rather than being spelled again at each place that
switches on it.
Optional throughout: a backend that provides none of it satisfies the
interface exactly as before.
… accepts
The previous version of this branch said nothing behaved differently. Four
things did.
`COMPUTER_USE_SEMANTIC_ACTIONS` declared `launch_app`, `scroll_element`,
`element_sequence` and `window_action`. None of those is in the tool's wire
enum on main; they exist only on the branch that adds the executor. The
constant feeds `APPROVAL_ACTIONS`, so `computerUseApprovalSummary` flipped
`knownAction` to true for four names the SDK rejects before the tool ever
runs, and `rememberForTurnAllowed` could evaluate true for two of them. A
person would have read an approval for an action that did not happen. The
constant is now scoped to what this schema accepts. Declaring an executor's
vocabulary is worth doing on the branch that also ships the executor.
`dispatch_refused` had no producer, and adding it to `COMPUTER_USE_ERROR_CODES`
widened `isComputerUseErrorCode` — the filter that decides whether a raw string
from the driver process is passed through or replaced with `capture_failed`.
That is a live trust boundary, not a declaration. Removed until something
produces it. The unproduced fields on the observation and outcome types are
left as they are: they have no runtime effect, and declaring the shape the
executor returns is what this branch is for.
The drift guard the constant's rationale rests on did not exist. The comment
promised a file that was never written, and the guard on main only walks from
the schemas to the catalog, which is the direction that catches a missing name
— not the direction this branch broke. `computer-use-schema-parity.test.ts`
now compares the two in both directions, with a negative control that runs the
real comparator over a fixed catalog so it keeps proving something on the day
the real one is correct.
`ComputerUseSettings` declared `enabled: false` and was registered nowhere:
not in `AppSettings`, not in `UpdateAppSettingsInput`, not in the defaults,
merge or normalize passes that every peer settings module goes through. In the
shipped app Computer Use tools are assembled with no enable check at all, so
the type told a reader the capability is off by default while it is always on.
Dropped. Wiring a real switch is a change with a UI and a migration, and it
does not belong under a types commit.
The model-call projection could not be used. Only `COMPUTER_USE_SEMANTIC_ACTIONS`
was exported from the package entry, so `computerUseModelCallArgs` — the
headline fix — was unreachable from `@maka/runtime`, and the defective
projection it exists to replace stayed in place. It is exported now and wired
into `projectPiToolArgs`, which is what writes the persisted `tool_call`
message and the `tool_start` event: the record the model reads back as its own
call. It had been reading the host's approval summary, which renames
`window_id` to `windowId`, adds two fields the model never sent, and drops
every argument it does not need.
And the projection withheld the wrong half of `text`. It keyed on argument
name, but `text` is six arguments under one name: the key for `press_key`,
`key` and `hold_key`, the element action name for `secondary_action`, a
substring of the window for `select_text`, and typed content for `type`.
Excluding all six is right for two of them. For the rest the model was reading
back `press_key ... text: <text>` and could not see which key it had pressed —
the exact failure the comment cited as motivation. It now keys on the action.
An action the tool does not accept has no plain set, so it falls through to
shapes rather than to whichever set was used last.
…ations nothing produces
The projection withheld coordinates on the grounds that they are
screen-derived. They are not: a coordinate is the model's own output,
four digits it chose and sent. Reduced to `<point>`, a model that clicked
[412, 88] and missed could not tell whether it had already tried that
point — the repeated-call shape this projection exists to make visible,
reintroduced by the projection itself. `coordinate`, `start_coordinate`
and `region` now come back whole when they are integers, and still
degrade to a shape when they are not.
An action the schema does not carry was recorded as `unknown`, so a model
whose call was rejected for the name it sent read its own history as a
call it never made and could not connect the rejection to the cause.
Collapsing an unrecognised name is `computerUseApprovalSummary`'s job,
where `knownAction` decides what a person is asked to allow; this record
is the model's own arguments, and the action is one of them. It is
bounded and redacted like every other model-supplied string.
The rest of the branch declared fourteen fields, one interface, one
backend method and two union members that nothing in the repository sets
and nothing reads. The parity guard added here covers the action-name
catalog and nothing else, so those declarations had nothing holding them
honest — which is the same condition the guard's own header describes:
`window_action` sat in the strict union and not on the wire and was
unreachable from the day it shipped, and nothing failed. This branch was
putting `window_action` back in that state, along with `scroll_element`.
They come with the executor that produces them, where a guard can check
that they arrive.
What is left is what has a consumer: `COMPUTER_USE_SEMANTIC_ACTIONS` with
the two-way parity check that keeps it and the wire enum in step, and
`computerUseModelCallArgs` wired into the Pi backend's tool_call record.
The projection was wired into PiAgentBackend, which the desktop app never
instantiates: boot.ts registers only ai-sdk and fake, and the pi backend
is reachable only from harbor-cell.ts behind MAKA_PI_*, driving an
external CLI, and it does not build a ToolRuntime at all. tool-runtime.ts
had a zero-byte diff, so on the shipping path nothing changed.
Measured on the real ToolRuntime before this commit, for a model that
sent {action:'press_key', app:'TextEdit', window_id:41,
observation_id:'obs-7', element_id:'e12', text:'cmd+s'}, the persisted
tool_start args were {action, approvalClass, rememberForTurnAllowed, app,
windowId:41, observationId} — a key the tool rejects, two host fields the
model never sent, no element, and a press_key with no key. model-history
replays event.content.args, so that is what the model read back of its
own call, and it went on calling it that way.
The line moves; the projection does not. The permission prompt still
reads permissionArgs and the approval scope key is still computed from
the raw call, so only what is written down changes.
Two things now cross the privacy boundary that did not, and
computer-use-privacy-boundary.test.ts says so where it asserts them.
element_id: an index the model chose into an observation it had already
read. Admitted only when it is a stable identifier, so an accessibility
label arriving under that key is dropped rather than persisted. Without
it every element action in a turn reads the same.
coordinate, start_coordinate, region: geometry the model wrote into the
call itself, integers only, so a mistyped value still degrades to a
shape. A coordinate is not read off the screen; withholding it left a
model that clicked a point and missed unable to see that it had already
tried that point.
What still does not cross: the value of text for type and select_text,
the value of set_value, and every other argument whose value is screen
content or something a person asked to have typed — those keep their key
and lose their value to <text>. text for press_key, key, hold_key and
secondary_action is a name from a closed set the executor publishes, so
it is carried; that is one argument name meaning six things, and only two
of the six come from outside the model.
element_identity joins approvalClass and rememberForTurnAllowed as
host-only. The Computer Use tool's own permissionArgs resolves the
model's element_id against the live observation and attaches it, and
permissionArgs is what this projection now runs on, so without that the
model would read back a key it cannot send whose value came off the
accessibility tree.
The new assertion is on the tool_start event, not on the projection
function. A test that calls computerUseModelCallArgs directly stays green
while the runtime writes the approval summary, which is how this survived
being fixed once already.
@hqhq1025
hqhq1025force-pushed the pr/cu-capability-types branch from 03fe3d9 to 71df4c1CompareAugust 3, 2026 11:48
@hqhq1025
hqhq1025 merged commit 80eb37d into apache:mainAug 3, 2026
10 checks passed
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit to hqhq1025/maka-agent that referenced this pull request Aug 3, 2026
…e now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. apache#1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after apache#1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
hqhq1025 added a commit that referenced this pull request Aug 3, 2026
…1888)
* feat(ui): give driving the computer its own row, and say what it did
Computer Use's display name is a noun — "Maka Computer" — so a turn that
observed a window, clicked a button and observed again rendered three
identical rows. `computerActionLabel` derives a row label from the call's
own arguments instead: 观察「计算器」窗口, 点击元素 e7.
The label is derived, never declared. The model is not given an `intent`
field to write; every word it produces is owned by the runtime, and a free
text field would be one more place it can be wrong.
What the renderer can read is `computerUseApprovalSummary(...)`, which
`ToolRuntime.executeTool` substitutes for the wire call before anything is
persisted. `element_id` had to be carried on that projection for a row to
name its target at all, and this adds it — but a shape filter is not a
privacy boundary. `stableIdentifier` admits `[A-Za-z0-9._:-]{1,256}`, which
is also the shape of an API key, and arguments are not validated before the
projection runs, so a model that put a token under `element_id` wrote it
verbatim into the persisted `tool_call`, onto `tool_start` and into the row
— while the same string under `observation_id` came out `[redacted]`. It
now takes the same `redactSecrets` pass `observationId` and `app` do.
Also puts `'computer'` on the wire as a tool activity kind and has the
Session decoder read the one list rather than a second hand-written copy of
it, which had already fallen behind and rejected the new kind outright.
Note for clients behind the runtime-host Session protocol: the `tool_start`
frame carries no `args`, so such a client gets the kind but cannot derive
the label. The desktop receives full SessionEvents over IPC and is
unaffected.
* fix(ui): read the Computer Use row's arguments in the dialect they are now persisted in
This branch was written when `ToolRuntime` persisted a Computer Use call as
`computerUseApprovalSummary(...)`, which spells the target `windowId` and
`elementId`. #1891 changed that line to `computerUseModelCallArgs(...)`, which
spells the same two fields `window_id` and `element_id`, because the model reads
its own history back through those args and was copying a dialect the tool
rejects.
Reading the old names off the new projection is not a crash and not a type
error. Every element action falls back to the generic "点击该元素" and every
window-scoped observe loses its window — which is the exact defect this branch
exists to remove, reintroduced by the rebase and invisible from either
renderer's own suite, because both build their fixtures by calling a projection
and neither could tell which one the runtime uses.
- `computer-action-label.ts` and `pi-transcript-tools.ts` read `window_id` and
`element_id`. The label module's key type now filters the projection's index
signature back out, so a stale name is a build error rather than an
`undefined`; before, `keyof` widened to `string` and accepted anything.
- Both renderers' fixtures go through `computerUseModelCallArgs`, and the seam
test says in one place which object is the dialect and which two files read
it.
- The row refuses to print an element id that is not identifier-shaped. The
persisted record keeps whatever the model sent, deliberately — it has to read
back the call it made — but free text under that key is either a label copied
off the screen or a mistake, and neither belongs in a sentence a person reads.
The redaction this branch carries is unchanged in effect and now lives where the
value actually lands: `computerUseModelCallArgs` already runs `element_id`
through `redactSecrets`, and the seam test pins it — an `element_id` of
`sk-ant-api03-…` is persisted as `[redacted]`, matching `app`. The addition of
`elementId` to `ComputerUseApprovalSummary` is dropped: after #1891 that
projection is neither persisted nor rendered, and its only remaining caller
reads `approvalClass`, so the field would have had no producer and no reader.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@hqhq1025