Run the input NIFs on a dirty scheduler instead of stalling the device

CHANGELOG.md

@@ -10,6 +10,38 @@ Full module documentation: [hexdocs.pm/mob](https://hexdocs.pm/mob).
 
 ## [Unreleased]
 
+### Changed
+- **Input NIFs now run on a dirty IO scheduler.** `tap`, `tap_xy`,
+ `long_press_xy`, `swipe_xy`, `type_text`, `delete_backward` and `clear_text`
+ on both platforms, plus `key_press`, `ax_action` and `ax_action_at_xy` on
+ iOS, are registered `ERL_NIF_DIRTY_JOB_IO_BOUND`. Every one of them blocks
+ waiting on the platform UI thread, and Android runs with a single normal
+ scheduler (`-S 1:1`), so an 800ms `long_press_xy` was stopping every process
+ on the device — timers, renders, `:rpc`, PubSub — for the duration. iOS's
+ `nif_long_press_xy` had been sleeping the caller's full duration on a normal
+ scheduler since it was written, and `nif_ax_action_at_xy` sleeps up to
+ 4 x 50ms retrying its lookup.
+
+ **Rebuild native** to pick this up (`mix mob.deploy --native`): the NIF
+ registration table is compiled into each app. See
+ `decisions/2026-09-05-input-nifs-are-dirty-io.md`.
+
+### Documented
+- **Android synthetic input in `Mob.Test`.** The platform matrix now covers
+ `long_press_xy/4`, `type_text/2`, `delete_backward/1` and `clear_text/1`, and
+ marks `tap_xy/3` and `swipe/5` as working on Android — for apps generated by
+ `mob_new` 0.4.32 or newer, since the methods live in the app's own generated
+ bridge. The new ⊕ footnote covers what that costs: gestures block for their
+ real duration, only the activity's own window is reachable (not dialogs or
+ modal sheets), `type_text/2` is ASCII-only and rejects a whole string
+ containing one unmappable character, and `clear_text/1` is deliberately
+ absent on Android rather than broken.
+- **`scroll_info/2` returns device pixels**, while `element_frames/1` returns
+ dp and `tap_xy/3` and `swipe/5` take dp. Feeding one into the other
+ overshoots by the display density.
+- The `✱` footnote no longer attributes `type_text/2`, `delete_backward/1` and
+ `clear_text/1` to the IOHID injection path they do not use.
+
 ### Added
 - **`Mob.Test.capabilities/1`** — ask a build which test-harness probes it can
  actually serve, before choosing how to drive it. Which probes work is a

android/jni/mob_nif.zig

@@ -4262,14 +4262,16 @@ const nif_funcs = [_]erts.ErlNifFunc{
  .{ .name = "ax_action_at_xy", .arity = 3, .fptr = nif_ax_action_at_xy, .flags = 0 },
  .{ .name = "ui_debug", .arity = 0, .fptr = nif_ui_debug, .flags = erts.ERL_NIF_DIRTY_JOB_CPU_BOUND },
  .{ .name = "screen_info", .arity = 0, .fptr = nif_screen_info, .flags = 0 },
- .{ .name = "tap", .arity = 1, .fptr = nif_tap, .flags = 0 },
- .{ .name = "tap_xy", .arity = 2, .fptr = nif_tap_xy, .flags = 0 },
- .{ .name = "type_text", .arity = 1, .fptr = nif_type_text, .flags = 0 },
- .{ .name = "delete_backward", .arity = 0, .fptr = nif_delete_backward, .flags = 0 },
+ .{ .name = "tap", .arity = 1, .fptr = nif_tap, .flags = erts.ERL_NIF_DIRTY_JOB_IO_BOUND },
+ .{ .name = "tap_xy", .arity = 2, .fptr = nif_tap_xy, .flags = erts.ERL_NIF_DIRTY_JOB_IO_BOUND },
+ .{ .name = "type_text", .arity = 1, .fptr = nif_type_text, .flags = erts.ERL_NIF_DIRTY_JOB_IO_BOUND },
+ .{ .name = "delete_backward", .arity = 0, .fptr = nif_delete_backward, .flags = erts.ERL_NIF_DIRTY_JOB_IO_BOUND },
+ // Not dirty: this one is a stub that returns :not_implemented without
+ // touching the UI thread. Flag it when it grows a real body.
  .{ .name = "key_press", .arity = 1, .fptr = nif_key_press, .flags = 0 },
- .{ .name = "clear_text", .arity = 0, .fptr = nif_clear_text, .flags = 0 },
- .{ .name = "long_press_xy", .arity = 3, .fptr = nif_long_press_xy, .flags = 0 },
- .{ .name = "swipe_xy", .arity = 4, .fptr = nif_swipe_xy, .flags = 0 },
+ .{ .name = "clear_text", .arity = 0, .fptr = nif_clear_text, .flags = erts.ERL_NIF_DIRTY_JOB_IO_BOUND },
+ .{ .name = "long_press_xy", .arity = 3, .fptr = nif_long_press_xy, .flags = erts.ERL_NIF_DIRTY_JOB_IO_BOUND },
+ .{ .name = "swipe_xy", .arity = 4, .fptr = nif_swipe_xy, .flags = erts.ERL_NIF_DIRTY_JOB_IO_BOUND },
  .{ .name = "screenshot", .arity = 3, .fptr = nif_screenshot, .flags = erts.ERL_NIF_DIRTY_JOB_CPU_BOUND },
  .{ .name = "scroll_info", .arity = 1, .fptr = nif_scroll_info, .flags = 0 },
  .{ .name = "scroll_to", .arity = 3, .fptr = nif_scroll_to, .flags = 0 },

decisions/2026-09-05-input-nifs-are-dirty-io.md

@@ -0,0 +1,87 @@
+# Harness input NIFs run on a dirty IO scheduler
+
+Date: 2026-09-05
+Status: accepted
+Ticket: MOB-160
+
+## Context
+
+Android's default BEAM argv is `-S 1:1 -SDcpu 1:1 -SDio 1 -A 1`
+(`android/jni/mob_beam.zig`). One normal scheduler. Whatever blocks it blocks
+every process on the device.
+
+Every harness input NIF blocks. They hand work to the platform UI thread and
+wait for the answer: Android through a `CountDownLatch` against a main-thread
+coroutine, iOS through `dispatch_sync` to the main queue. Until now all of
+them except iOS's `tap_xy` were registered with `.flags = 0`.
+
+MOB-160 made this acute rather than theoretical. Android gestures now have to
+hold the pointer for their real duration, because the platform's long-press
+and drag detectors wait on posted callbacks and frame boundaries and ignore
+synthesised timestamps. `long_press_xy(node, x, y, 800)` blocks for 800ms.
+iOS was already there and nobody noticed: on the device branch
+`nif_long_press_xy` calls `[NSThread sleepForTimeInterval:]` for the caller's
+full duration on a normal scheduler. (The simulator branch drives the press
+through `_setState:` instead and does not sleep. The device sleep is doubly
+wasteful, since per the 2026-08-09 decision that injection path is accepted
+and never delivered — it is 800ms of dead time.)
+
+800ms is roughly 800 times the ~1ms budget the Erlang efficiency guide gives a
+NIF. For that window the device does nothing at all: no timers, no renders, no
+`:rpc`, no PubSub. An agent driving the UI would be pausing the app it is
+trying to observe.
+
+## Decision
+
+`tap`, `tap_xy`, `long_press_xy`, `swipe_xy`, `type_text`, `delete_backward`
+and `clear_text` are registered `ERL_NIF_DIRTY_JOB_IO_BOUND` on both platforms.
+iOS additionally flags `key_press`, `ax_action` and `ax_action_at_xy`.
+
+(There is no `tap_by_label` NIF on either platform — it is a capability atom
+and a Kotlin method name; the NIF behind it is `tap/1`.)
+
+The flags diverge by platform because the code does. Android's `key_press` is
+a hardcoded `:not_implemented` stub that never touches the UI thread, so a
+dirty hop would buy nothing; iOS's `dispatch_sync`s. Android has no
+accessibility path at all, while iOS's `nif_ax_action_at_xy` retries its
+lookup four times with `[NSThread sleepForTimeInterval:0.05]` between
+attempts — up to ~150ms of literal sleep, which is the same bug in a place
+nobody was looking.
+
+IO-bound rather than CPU-bound: they are not computing, they are waiting on
+another thread, which is what that flag is for. The default argv already
+provisions a dirty IO scheduler (`-SDio 1`), so they get a thread that is not
+the one running everyone else's processes.
+
+This extends `decisions/2026-08-09-tap-xy-reports-observed-effect.md`, which
+moved iOS's `tap_xy` because it blocks for the settle window. That reasoning
+was right and was applied too narrowly — Android's copy of `tap_xy` never
+followed at all — but it does not simply generalise: only `tap_xy` has a settle
+window. What covers the rest is plainer, and was true the whole time: they
+block on `dispatch_sync` or on a latch.
+
+The previous note above `nif_funcs[]` in `ios/mob_nif.m` did state a principle,
+so this is a reversal rather than a gap. It held that the harness calls these
+in tight loops and that dirty-dispatch overhead would add up, pending
+benchmarks that were never run. That trade is the wrong way round: the cost is
+a thread wakeup per call (real — `-sbwtdio none` means the dirty scheduler
+does not busy-wait, so each call pays one), and the thing being traded away is
+the whole VM for the duration. We have not measured the wakeup cost, and are
+accepting it deliberately rather than claiming it is free.
+
+## Consequences
+
+- A blocking input NIF costs a dirty IO scheduler slot, not the VM.
+- There is exactly one dirty IO scheduler (`-SDio 1`), so this converts a
+ total stall into head-of-line blocking on a resource of size one. An 800ms
+ long press now delays anything else that is IO-dirty — on Android that
+ includes `resolve_ipv4` (the DNS path), `audio_output_level` and
+ `vendor_usb_bulk_write`; on iOS, `safe_area`, which is on a layout path.
+ Strictly better than what it replaces, but it is a new contention edge, and
+ the answer if it bites is to raise `-SDio`, not to go back.
+- `long_press_xy` still takes its full duration. That is inherent — a long
+ press that returns early is not a long press. The call is now merely slow
+ rather than globally stalling.
+- The rule to apply to anything added here later: **if it waits on the UI
+ thread, it is dirty.** The old table had no principle, which is how one
+ member of a group of nine ended up flagged correctly and eight did not.

ios/mob_nif.m

@@ -7969,15 +7969,21 @@ static ERL_NIF_TERM nif_vendor_usb_close(ErlNifEnv *env, int argc, const ERL_NIF
 // * ui_tree — recursive UIAccessibility walk (variable, can be 10s of ms)
 // * ui_debug — same walk, more output
 //
-// Synthetic-input NIFs (swipe_xy, long_press_xy, type_text, key_press,
-// delete_backward, clear_text) dispatch_sync to the main queue but also do
-// some pre-dispatch work; they're left on regular schedulers for now because
-// the test harness calls them in tight loops and dirty-dispatch overhead would
-// add up. Re-evaluate if benchmarks show scheduler stalls under heavy harness use.
+// The input NIFs (tap, tap_xy, swipe_xy, long_press_xy, type_text, key_press,
+// delete_backward, clear_text, ax_action, ax_action_at_xy) are all
+// ERL_NIF_DIRTY_JOB_IO_BOUND: every one of them blocks a scheduler waiting on
+// the main queue. See decisions/2026-09-05-input-nifs-are-dirty-io.md.
 //
-// tap_xy is the exception: it blocks up to MOB_TAP_SETTLE_MS waiting for the
-// app to react (that wait is what makes its :ok trustworthy), which is far too
-// long to hold a normal scheduler.
+// This reverses an earlier note here, which kept them on regular schedulers on
+// the grounds that the harness calls them in tight loops and dirty-dispatch
+// overhead would add up, pending benchmarks. That trade was the wrong way
+// round: the overhead is a thread wakeup, while the stall is the whole VM.
+// nif_long_press_xy sleeps the caller's full duration on the device branch,
+// and nif_ax_action_at_xy sleeps up to 4 x 50ms retrying its lookup — on
+// Android there is exactly one normal scheduler, so that is the entire device.
+//
+// The rule for anything added below: if it waits on the main queue, it is
+// dirty. IO_BOUND rather than CPU_BOUND, because it is waiting, not computing.
 static ErlNifFunc nif_funcs[] = {
 #if !MOB_RELEASE
  // ── Test harness (listed first to survive linker dead-code stripping) ──────
@@ -7990,16 +7996,16 @@ static ERL_NIF_TERM nif_vendor_usb_close(ErlNifEnv *env, int argc, const ERL_NIF
  {"ui_paint_debug", 0, nif_ui_paint_debug, ERL_NIF_DIRTY_JOB_CPU_BOUND},
  {"ui_debug", 0, nif_ui_debug, ERL_NIF_DIRTY_JOB_CPU_BOUND},
  {"screen_info", 0, nif_screen_info, 0},
- {"tap", 1, nif_tap, 0},
- {"ax_action", 2, nif_ax_action, 0},
- {"ax_action_at_xy", 3, nif_ax_action_at_xy, 0},
+ {"tap", 1, nif_tap, ERL_NIF_DIRTY_JOB_IO_BOUND},
+ {"ax_action", 2, nif_ax_action, ERL_NIF_DIRTY_JOB_IO_BOUND},
+ {"ax_action_at_xy", 3, nif_ax_action_at_xy, ERL_NIF_DIRTY_JOB_IO_BOUND},
  {"tap_xy", 2, nif_tap_xy, ERL_NIF_DIRTY_JOB_IO_BOUND},
- {"type_text", 1, nif_type_text, 0},
- {"delete_backward", 0, nif_delete_backward, 0},
- {"key_press", 1, nif_key_press, 0},
- {"clear_text", 0, nif_clear_text, 0},
- {"long_press_xy", 3, nif_long_press_xy, 0},
- {"swipe_xy", 4, nif_swipe_xy, 0},
+ {"type_text", 1, nif_type_text, ERL_NIF_DIRTY_JOB_IO_BOUND},
+ {"delete_backward", 0, nif_delete_backward, ERL_NIF_DIRTY_JOB_IO_BOUND},
+ {"key_press", 1, nif_key_press, ERL_NIF_DIRTY_JOB_IO_BOUND},
+ {"clear_text", 0, nif_clear_text, ERL_NIF_DIRTY_JOB_IO_BOUND},
+ {"long_press_xy", 3, nif_long_press_xy, ERL_NIF_DIRTY_JOB_IO_BOUND},
+ {"swipe_xy", 4, nif_swipe_xy, ERL_NIF_DIRTY_JOB_IO_BOUND},
 #endif
  {"capabilities", 0, nif_capabilities, 0},
 #if !MOB_RELEASE || defined(MOB_ENABLE_SCREENSHOT)

lib/mob/test.ex

@@ -132,18 +132,22 @@ defmodule Mob.Test do
  | `toggle/2` | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
  | `dismiss_alert/2` | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
  | `adjust_slider/4` | ⚠️ AX active§ | ⚠️ AX active§ | ❌ ui_tree_unavailable |
- | `tap_xy/3` | ⚠️ AX-activatable only¶ | ❌ no_effect¶ | n/a |
- | `swipe/5` | ⚠️ scroll only| ⚠️ unverified✱| n/a |
+ | `tap_xy/3` | ⚠️ AX-activatable only¶ | ❌ no_effect¶ | ✅ ⊕ |
+ | `long_press_xy/4` | ⚠️ acceptance only✱| ⚠️ acceptance only✱| ✅ ⊕ |
+ | `swipe/5` | ⚠️ scroll only| ⚠️ acceptance only✱| ✅ ⊕ |
+ | `type_text/2` | ⚠️ acceptance only✱| ⚠️ acceptance only✱| ✅ ASCII only⊕ |
+ | `delete_backward/1` | ⚠️ acceptance only✱| ⚠️ acceptance only✱| ✅ ⊕ |
+ | `clear_text/1` | ⚠️ acceptance only✱| ⚠️ acceptance only✱| ❌ not_loaded⊕ |
  | `capabilities/1` | ✅ | ✅ | ✅ |
 
  **This table is a snapshot, and snapshots drift.** Ask the running app
  instead — `capabilities/1` reports what THIS build can actually serve. On
  Android that is a per-*app* fact, since each harness NIF bails when its
  cached `MobBridge` method is absent and the bridge is generated once and
  never re-rendered; on iOS it is per-*configuration*, since the whole harness
- is compiled out of release builds. Measured on a freshly generated Android
-app, `tap_xy` and `type_text` are both unavailable (MOB-160) — the `n/a` in
-the rows above reads as "not applicable" but means "no bridge method".
+ is compiled out of release builds. Apps generated before `mob_new` 0.4.32
+have no synthetic-input methods in their bridge at all and report `false` for
+every one of them; regenerate with a new enough `mob_new` to pick them up.
 
  - **†** SwiftUI doesn't expose its content as separate UIView instances —
  `view_tree` reaches the SwiftUI hosting view's container and stops.
@@ -172,9 +176,54 @@ defmodule Mob.Test do
  coordinate returns `{:error, :no_effect}`. Drive taps with `tap/2` (by tag);
  see `tap_xy/3` and
  `decisions/2026-08-09-ios-device-tap-injection-has-no-effect.md`.
- - **✱** `swipe/5` and `long_press/4` use the same device injection path as
- `tap_xy/3` and still report `:ok` on acceptance rather than on effect.
- Same root cause, not yet converted — treat their `:ok` as unverified.
+ - **⊕** Android synthesises input in-process, dispatching `MotionEvent`s at
+ the activity's decor view and `KeyEvent`s at the activity. No `adb`, no
+ `INJECT_EVENTS` (a signature permission no app can hold).
+
+ **The ✅ is a property of the app, not of mob.** These call methods on the
+ app's own generated `MobBridge`, which ships in `mob_new` — they work in an
+ app generated by `mob_new` 0.4.32 or newer, and return
+ `{:error, :not_loaded}` in every app generated before that, however new the
+ `mob` it runs. `MobBridge.kt` is generated once and never re-rendered, so
+ an existing app needs regenerating. `capabilities/1` answers this for the
+ build in front of you; the table cannot.
+
+ Four more consequences worth knowing before you rely on it:
+
+ * **Gestures cost real wall-clock.** A long press or swipe has to hold
+ the pointer for its real duration, because Android's detectors wait on
+ posted callbacks and frame boundaries — synthesised timestamps are
+ ignored. `long_press_xy(node, x, y, 800)` blocks for 800ms. These NIFs
+ run on a dirty IO scheduler for that reason.
+ * **Only the activity's own window is reachable.** A `Dialog` or a
+ Material `ModalBottomSheet` renders in its own window, so a tap aimed
+ at one lands on the dimmed activity behind it.
+ * **`type_text/2` is ASCII-only.** The virtual keyboard has no key
+ sequence for emoji or accented Latin, and one unmappable character
+ rejects the whole string, so nothing is typed.
+ * **`clear_text/1` is absent, not broken.** Two implementations reported
+ success while clearing nothing (events coalesce faster than the field
+ recomposes), so the bridge ships without it and the call returns
+ `{:error, :not_loaded}`. Select-all-and-delete by hand, or rebuild the
+ field's state through your own event.
+
+ Verified on a physical device: tap navigates, long press fires
+ `on_long_press`, swipe scrolls a scroll view, typing and backspace change
+ the field. `on_long_press` fires on `column`, `row`, `text`, `icon` and
+ `box` — not `button`, matching iOS — so a long press on a `button` does
+ nothing by design.
+
+ - **✱** These report `:ok` when the OS *accepted* the input, not when the app
+ was observed to react — they have not been converted to the observation
+ model `tap_xy/3` uses. Treat their `:ok` as "sent", not "worked".
+
+ Two different reasons sit behind that, and they lead to different bugs.
+ `swipe/5` and `long_press_xy/4` ride the same IOHID injection path as
+ `tap_xy/3`, which on a physical device is accepted and never delivered —
+ so their `:ok` there is actively misleading. `type_text/2`,
+ `delete_backward/1` and `clear_text/1` do NOT touch that path; they
+ `dispatch_sync` and message the first responder directly, so they do
+ something real, and merely fail to confirm it.
 
  Helpers that depend on AX return clear error tuples on Android instead of
  raising. Callers should match on `{:error, :not_supported_on_android}` and
@@ -1197,7 +1246,14 @@ defmodule Mob.Test do
  :rpc.call(node, :mob_nif, :key_press, [key])
  end
 
- @doc "Clear all text in the focused input (select-all + delete)."
+ @doc """
+ Clear all text in the focused input (select-all + delete).
+
+ On Android this returns `{:error, :not_loaded}`: the generated bridge ships
+ without a `clearText` method on purpose, because both implementations tried
+ reported success while clearing nothing. See the `⊕` note on the platform
+ matrix above.
+ """
  @spec clear_text(node()) :: :ok | {:error, atom()}
  def clear_text(node) do
  :rpc.call(node, :mob_nif, :clear_text, [])
@@ -1379,6 +1435,14 @@ defmodule Mob.Test do
  Mob.Test.scroll_info(node, "feed")
  #=> %{offset: {0.0, 0.0}, content: {393.0, 2400.0}, viewport: {393.0, 756.0},
  # max_offset: {0.0, 1644.0}, kind: :pixel}
+
+ > #### These are device pixels, not dp {: .warning}
+ >
+ > `element_frames/1` returns dp, and `tap_xy/3` and `swipe/5` take dp. A
+ > `:pixel` scroll offset is in raw device pixels. Feeding one straight into
+ > the other overshoots by the display density — 2.75x on a moto g power.
+ > Divide by the density, or drive scrolling with `scroll_to/4`, which works
+ > in whatever unit `:kind` reports.
  """
  @spec scroll_info(node(), String.t() | atom()) :: map() | {:error, term()}
  def scroll_info(node, id) do