Give generated Android apps synthetic input (MOB-160) - #56
Merged
Conversation
Generated apps shipped no way to synthesise a tap. An agent driving one over
dist could read state but not touch it, so `Mob.Test.tap_xy/3` and friends
returned `{:error, :not_loaded}` on every Android app ever generated.
Adds tapXy, longPressXy, swipeXy, typeText and deleteBackward to the bridge
template, dispatching MotionEvents at the activity's decor view in-process.
The obvious route, Instrumentation.sendPointerSync, needs INJECT_EVENTS — a
signature permission no ordinary app can hold.
Verified on a physical moto g power, each one against observed app state
rather than a return code: tap navigates between screens, long press fires
on_long_press, swipe scrolls a scroll view the full 1033px, typing and
backspace change a focused field. That distinction earned its keep — every
method here returned :ok while doing nothing at some point in development.
Two findings worth recording:
Gestures must be dispatched over REAL elapsed time. The first implementation
built each gesture inside one main-thread block with fabricated timestamps.
Every event reached the view and every gesture was ignored: Android's
long-press detector waits on a posted callback, and Compose resolves drags in
a pointer-input coroutine that only resumes when the looper is free. A block
that runs a gesture to completion without yielding starves exactly the
machinery meant to interpret it.
clearText ships ABSENT rather than broken. Two implementations reported
success while clearing nothing — dispatched backspaces coalesce within a frame
(~4 of 200 registered), and Ctrl+A does not select in a Compose text field.
The JNI lookup is a cacheOptional, so an absent method leaves the handle null,
capabilities/1 reports clear_text: false, and calls return :not_loaded. An
agent can plan around a capability it knows it lacks; it cannot plan around a
lie.
Hardened after adversarial review, which caught the same sin shipped elsewhere
in this change:
- Results report what was CONSUMED, not what was sent. Discarding the
dispatch return made :dispatch_failed and :no_first_responder unreachable,
so typing into an unfocused field returned :ok. It now reports the error.
- ACTION_CANCEL on any failure between DOWN and UP. A dangling DOWN leaves the
view tree believing a finger is down, and every later touch in the session
reads as a second pointer, silently.
- A mutex across each gesture. These arrive as concurrent :rpc calls, and a
tap's DOWN landing inside a swipe's MOVEs corrupted both while both
reported success.
- The latch timeout is honoured rather than discarded, `ok` is an
AtomicBoolean, the job is cancelled on timeout, and InterruptedException is
caught — letting it propagate would unwind into JNI with a pending
exception, and those call sites do not ExceptionCheck.
- typeText gates on currentFocus and reports unmappable characters instead of
silently typing nothing.
- Refuses main-thread reentry, which would deadlock into an ANR.
Tests assert the methods exist, are @JvmStatic, and that each Kotlin signature
matches the JNI descriptor mob_nif.zig caches it with — a mismatch is not a
compile error, it is a null JMethodID and a silent :not_loaded on device. Also
asserts clearText stays absent, so the deliberate omission is not "fixed" in
good faith later. Mutation-checked: flipping durationMs to Int fails the
signature test.
Needs the matching mob change — these NIFs block a scheduler for the
gesture's duration and must run dirty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>Uh oh!
There was an error while loading. Please reload this page.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated Android apps shipped no way to synthesise a tap. An agent driving one over dist could read state but not touch it, so
Mob.Test.tap_xy/3and friends returned{:error, :not_loaded}on every Android app ever generated.Adds
tapXy,longPressXy,swipeXy,typeTextanddeleteBackwardto the bridge template, dispatchingMotionEvents at the activity's decor view in-process. The obvious route,Instrumentation.sendPointerSync, needsINJECT_EVENTS— a signature permission no ordinary app can hold.Verified on a physical device
moto g power (2021), each assertion against observed app state rather than a return code:
tap_xyHomeScreen→TextScreenby coordinatelong_press_xyon_long_presscounter 0 → 1swipe_xytype_text""→"abc";{:error, :no_first_responder}unfocuseddelete_backwardclear_text{:error, :not_loaded}That distinction earned its keep — every method here returned
:okwhile doing nothing at some point during development.Two findings worth reading
Gestures must be dispatched over real elapsed time. The first implementation built each gesture inside one main-thread block with fabricated timestamps. Every event reached the view and every gesture was ignored: Android's long-press detector waits on a posted callback, and Compose resolves drags in a pointer-input coroutine that only resumes when the looper is free. A block that runs a gesture to completion without yielding starves exactly the machinery meant to interpret it.
clearTextships absent rather than broken. Two implementations reported success while clearing nothing — dispatched backspaces coalesce within a frame (~4 of 200 registered), and Ctrl+A does not select in a Compose text field. The JNI lookup is acacheOptional, so an absent method leaves the handle null,capabilities/1reportsclear_text: false, and calls return:not_loaded. An agent can plan around a capability it knows it lacks; it cannot plan around a lie.Hardened after adversarial review
The review caught the same sin shipped elsewhere in this change: results were reporting success unconditionally.
:dispatch_failedand:no_first_responderunreachable, so typing into an unfocused field returned:ok.ACTION_CANCELon any failure between DOWN and UP. A dangling DOWN leaves the view tree believing a finger is down, and every later touch in the session reads as a second pointer, silently.:rpccalls, and a tap's DOWN landing inside a swipe's MOVEs corrupted both while both reported success.okis anAtomicBoolean, the job is cancelled on timeout, andInterruptedExceptionis caught — letting it propagate would unwind into JNI with a pending exception, and those call sites do notExceptionCheck.Known limits, all documented in
Mob.TestGestures block for their real duration; only the activity's own window is reachable (not dialogs or modal sheets);
typeTextis ASCII-only and rejects a whole string containing one unmappable character.MobBridge.ktis generated once and never re-rendered, so existing apps must be regenerated to pick this up.Tests
Assert the methods exist, are
@JvmStatic, and that each Kotlin signature matches the JNI descriptormob_nif.zigcaches it with — a mismatch is not a compile error, it is a nullJMethodIDand a silent:not_loadedon device. Also assertsclearTextstays absent so the deliberate omission is not "fixed" in good faith later. Mutation-checked: flippingdurationMstoIntfails the signature test.Suite: 411/413, the 2 failures pre-existing
MOB_DIRenv ones. ktlint clean.Requires
GenericJam/mob#135 — these NIFs block a scheduler for the gesture's duration and must run dirty.
🤖 Generated with Claude Code