Skip to content

Give generated Android apps synthetic input (MOB-160) - #56

Merged
GenericJam merged 1 commit into
masterfrom
fix/mob-160-android-input-bridge
Sep 6, 2026
Merged

Give generated Android apps synthetic input (MOB-160)#56
GenericJam merged 1 commit into
masterfrom
fix/mob-160-android-input-bridge

Conversation

@GenericJam

@GenericJamGenericJam commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Generated Android apps shipped no way to synthesise a tap. An agent driving one over dist could read state but not touch it, so Mob.Test.tap_xy/3 and friends returned {:error, :not_loaded} on every Android app ever generated.

Adds tapXy, longPressXy, swipeXy, typeText and deleteBackward to the bridge template, dispatching MotionEvents at the activity's decor view in-process. The obvious route, Instrumentation.sendPointerSync, needs INJECT_EVENTS — a signature permission no ordinary app can hold.

Verified on a physical device

moto g power (2021), each assertion against observed app state rather than a return code:

primitiveevidence
tap_xynavigated HomeScreenTextScreen by coordinate
long_press_xyon_long_press counter 0 → 1
swipe_xyscroll offset 0 → 1033px (full extent)
type_textfocused field """abc"; {:error, :no_first_responder} unfocused
delete_backwardremoved exactly one character
clear_textabsent by design, returns {:error, :not_loaded}

That distinction earned its keep — every method here returned :ok while doing nothing at some point during development.

Two findings worth reading

Gestures must be dispatched over real elapsed time. The first implementation built each gesture inside one main-thread block with fabricated timestamps. Every event reached the view and every gesture was ignored: Android's long-press detector waits on a posted callback, and Compose resolves drags in a pointer-input coroutine that only resumes when the looper is free. A block that runs a gesture to completion without yielding starves exactly the machinery meant to interpret it.

clearText ships absent rather than broken. Two implementations reported success while clearing nothing — dispatched backspaces coalesce within a frame (~4 of 200 registered), and Ctrl+A does not select in a Compose text field. The JNI lookup is a cacheOptional, so an absent method leaves the handle null, capabilities/1 reports clear_text: false, and calls return :not_loaded. An agent can plan around a capability it knows it lacks; it cannot plan around a lie.

Hardened after adversarial review

The review caught the same sin shipped elsewhere in this change: results were reporting success unconditionally.

  • Results report what was consumed, not what was sent. Discarding the dispatch return made :dispatch_failed and :no_first_responder unreachable, so typing into an unfocused field returned :ok.
  • ACTION_CANCEL on any failure between DOWN and UP. A dangling DOWN leaves the view tree believing a finger is down, and every later touch in the session reads as a second pointer, silently.
  • A mutex across each gesture — these arrive as concurrent :rpc calls, and a tap's DOWN landing inside a swipe's MOVEs corrupted both while both reported success.
  • The latch timeout is honoured rather than discarded, ok is an AtomicBoolean, the job is cancelled on timeout, and InterruptedException is caught — letting it propagate would unwind into JNI with a pending exception, and those call sites do not ExceptionCheck.
  • Refuses main-thread reentry, which would deadlock into an ANR.

Known limits, all documented in Mob.Test

Gestures block for their real duration; only the activity's own window is reachable (not dialogs or modal sheets); typeText is ASCII-only and rejects a whole string containing one unmappable character.

MobBridge.kt is generated once and never re-rendered, so existing apps must be regenerated to pick this up.

Tests

Assert the methods exist, are @JvmStatic, and that each Kotlin signature matches the JNI descriptor mob_nif.zig caches it with — a mismatch is not a compile error, it is a null JMethodID and a silent :not_loaded on device. Also asserts clearText stays absent so the deliberate omission is not "fixed" in good faith later. Mutation-checked: flipping durationMs to Int fails the signature test.

Suite: 411/413, the 2 failures pre-existing MOB_DIR env ones. ktlint clean.

Requires

GenericJam/mob#135 — these NIFs block a scheduler for the gesture's duration and must run dirty.

🤖 Generated with Claude Code

Generated apps shipped no way to synthesise a tap. An agent driving one over
dist could read state but not touch it, so `Mob.Test.tap_xy/3` and friends
returned `{:error, :not_loaded}` on every Android app ever generated.
Adds tapXy, longPressXy, swipeXy, typeText and deleteBackward to the bridge
template, dispatching MotionEvents at the activity's decor view in-process.
The obvious route, Instrumentation.sendPointerSync, needs INJECT_EVENTS — a
signature permission no ordinary app can hold.
Verified on a physical moto g power, each one against observed app state
rather than a return code: tap navigates between screens, long press fires
on_long_press, swipe scrolls a scroll view the full 1033px, typing and
backspace change a focused field. That distinction earned its keep — every
method here returned :ok while doing nothing at some point in development.
Two findings worth recording:
Gestures must be dispatched over REAL elapsed time. The first implementation
built each gesture inside one main-thread block with fabricated timestamps.
Every event reached the view and every gesture was ignored: Android's
long-press detector waits on a posted callback, and Compose resolves drags in
a pointer-input coroutine that only resumes when the looper is free. A block
that runs a gesture to completion without yielding starves exactly the
machinery meant to interpret it.
clearText ships ABSENT rather than broken. Two implementations reported
success while clearing nothing — dispatched backspaces coalesce within a frame
(~4 of 200 registered), and Ctrl+A does not select in a Compose text field.
The JNI lookup is a cacheOptional, so an absent method leaves the handle null,
capabilities/1 reports clear_text: false, and calls return :not_loaded. An
agent can plan around a capability it knows it lacks; it cannot plan around a
lie.
Hardened after adversarial review, which caught the same sin shipped elsewhere
in this change:
- Results report what was CONSUMED, not what was sent. Discarding the
dispatch return made :dispatch_failed and :no_first_responder unreachable,
so typing into an unfocused field returned :ok. It now reports the error.
- ACTION_CANCEL on any failure between DOWN and UP. A dangling DOWN leaves the
view tree believing a finger is down, and every later touch in the session
reads as a second pointer, silently.
- A mutex across each gesture. These arrive as concurrent :rpc calls, and a
tap's DOWN landing inside a swipe's MOVEs corrupted both while both
reported success.
- The latch timeout is honoured rather than discarded, `ok` is an
AtomicBoolean, the job is cancelled on timeout, and InterruptedException is
caught — letting it propagate would unwind into JNI with a pending
exception, and those call sites do not ExceptionCheck.
- typeText gates on currentFocus and reports unmappable characters instead of
silently typing nothing.
- Refuses main-thread reentry, which would deadlock into an ANR.
Tests assert the methods exist, are @JvmStatic, and that each Kotlin signature
matches the JNI descriptor mob_nif.zig caches it with — a mismatch is not a
compile error, it is a null JMethodID and a silent :not_loaded on device. Also
asserts clearText stays absent, so the deliberate omission is not "fixed" in
good faith later. Mutation-checked: flipping durationMs to Int fails the
signature test.
Needs the matching mob change — these NIFs block a scheduler for the
gesture's duration and must run dirty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@GenericJam
GenericJam merged commit 8c50823 into masterSep 6, 2026
3 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@GenericJam