From 56f05ff11c7b61da789399c948f568a9e8f7bfb9 Mon Sep 17 00:00:00 2001 From: GenericJam Date: Sat, 5 Sep 2026 00:53:39 -0600 Subject: [PATCH] Tell agents to ask what a build can be probed with MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `Mob.Test.capabilities/1` shipped documented only in the `Mob.Test` moduledoc, which is the wrong place: the guide an agent reads before choosing how to drive an app is `agentic_coding.md`, and choosing before discovering a probe is unavailable is the entire problem it solves. An adversarial review found three things wrong with the first version. The example showed nine of sixteen keys behind a trailing `...`, and every hidden one was `false` — including `sample_region`, which is what `Mob.Test.sample_color/2` rides on and which has no Android NIF at all, so the call raises rather than returning an error tuple. That is the precise misread the feature was built to prevent, reintroduced by abridging the example that demonstrates it. All seventeen entries are printed now. It claimed `Mob.Test.tap/2` works "through the render tree". It does not — it sends `{:tap, tag}` straight to the screen process over dist. The conclusion was right and the mechanism was invented, and the invented mechanism implies a guarantee the function does not offer: this same guide says two sections later that `tap/2` returns `:ok` whether or not any screen matched. And it sold the feature partly on iOS release builds, without saying that an iOS release build drops `-name` entirely and so cannot answer at all — it reports `dist_rpc: false` with everything false, indistinguishable from a bad node name. The docstring says this; the guide had dropped exactly the sentence that stops a reader misdiagnosing. Also from the review: the section sat above the API it qualifies, so its code block referenced a `node` bound twenty lines later; it named three outcomes when `classify_capabilities/1` has four, omitting the one where `load_nif` failed and the fix is a native rebuild rather than the bridge; it stated "each harness NIF returns `{:error, :not_loaded}`" as a rule with two exceptions inside the same probe set; and it never said what to do with the answer, which is to drop to Layer 2 — two subsections below it. Adds the CHANGELOG entry the implementation PR skipped, records the load side effects in the docstring, and puts the question in the priming block that readers actually paste into their AGENTS.md. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 11 ++++++++ guides/agentic_coding.md | 59 +++++++++++++++++++++++++++++++++++++++- lib/mob/test.ex | 7 +++++ 3 files changed, 76 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ab33d45f..ce1cea76 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,17 @@ Full module documentation: [hexdocs.pm/mob](https://hexdocs.pm/mob). ## [Unreleased] +### Added +- **`Mob.Test.capabilities/1`** — ask a build which test-harness probes it can + actually serve, before choosing how to drive it. Which probes work is a + runtime fact: on Android each harness NIF bails when the app's generated + `MobBridge.kt` lacks the matching method, and that file is generated once and + never re-rendered; on iOS the harness is compiled out of release builds. + Returns `:unknown` per probe for an app predating `mob_nif:capabilities/0` + rather than guessing, and `false` everywhere with `dist_rpc: false` when + nothing answered. Backed by a new `mob_nif:capabilities/0` NIF on both + platforms. + ### Fixed - **A parked screen no longer keeps live resources running, and returning to one no longer breaks the frame registry** (MOB-147, MOB-145). MOB-129 keeps diff --git a/guides/agentic_coding.md b/guides/agentic_coding.md index 155ab702..1a6db373 100644 --- a/guides/agentic_coding.md +++ b/guides/agentic_coding.md @@ -175,6 +175,62 @@ or directly from an agent that can run shell commands, using: iex -S mix --eval 'IO.inspect Mob.Test.assigns(:"mob_demo_ios@127.0.0.1")' ``` +**Ask what this build can be probed with, before committing to an approach.** + +Which probes work is a runtime fact, not a property of the platform. On Android +most harness NIFs return `{:error, :not_loaded}` when the app's generated +`MobBridge.kt` lacks the matching method, and that file is generated once and +never re-rendered, so apps drift as the template moves. On iOS the harness is +compiled out of release builds. + +```elixir +Mob.Test.capabilities(node) +#=> %{ +#=> dist_rpc: true, +#=> view_tree: false, ui_tree: false, screen_info: true, +#=> tap_xy: false, tap_by_label: false, long_press_xy: false, +#=> swipe_xy: false, type_text: false, delete_backward: false, +#=> clear_text: false, ax_action: false, element_frames: true, +#=> scroll_info: true, scroll_to: true, sample_region: false, +#=> screenshot: true +#=> } +``` + +That is a **freshly generated Android app** — abridged only in layout, not in +content; every one of the seventeen keys is shown, because guessing at the rest +is exactly what goes wrong. The template defines `screenInfo`, `elementFrames`, +`screenshot`, `scrollInfo` and `scrollTo`, and nothing else in the harness set +(MOB-160). + +Two of those `false`s bite harder than they look: + +* `sample_region: false` means `Mob.Test.sample_color/2` is unavailable — there + is no Android NIF for it at all, so the call **raises** rather than returning + an error tuple. `screenshot: true` sitting next to it is not a substitute. +* `tap_by_label: false` removes the documented fallback for `tap_xy`, so this + build has no synthetic input of any kind. + +`Mob.Test.tap/2` still works: it delivers the same `{:tap, tag}` message a +native tap produces straight to the screen process over dist, never touching +the bridge. It is fire-and-forget — `:ok` whether or not a screen matched — so +assert the state change, per *The honesty contract* above. + +**When there is no synthetic input, that is your cue to drop to Layer 2**: drive +the UI with `mcp__adb__*` below and keep `Mob.Test` for reading state. + +Four answers, and three of them are not "the app is fine": + +| Result | Means | +|--------|-------| +| probes true/false, `dist_rpc: true` | the real answer | +| all `:unknown`, `dist_rpc: true` | app predates `capabilities/0` (added 0.7.40) — upgrade `mob` rather than guessing | +| all `false`, `dist_rpc: true` | `load_nif` failed on the device: every NIF is down, and the fix is a native rebuild, not the bridge | +| all `false`, `dist_rpc: false` | nothing answered. An iOS **release** build reports exactly this, because it drops `-name` and has no distribution at all — indistinguishable from a bad node name or a dead tunnel, so check which you expect | + +`capabilities/2` takes a timeout, defaulting to 5s. That matters here because +this is the first call an agent makes, and a wedged-but-reachable device would +otherwise hang it indefinitely. + #### Layer 2 — MCP platform tools (for rendering and layout) When the question is visual — "does this text overflow?", "is the button in the right @@ -410,7 +466,8 @@ screenshots or adb screencap as your primary inspection method. Instead: 1. Run `mix mob.connect --no-iex` to establish distribution tunnels (if not already running) 2. Use `Mob.Test` from IEx to query exact state: - - `Mob.Test.screen(node)` — what screen is active? + - `Mob.Test.capabilities(node) # ask FIRST: which probes does this build serve? +Mob.Test.screen(node)` — what screen is active? - `Mob.Test.assigns(node)` — what is the live data? - `Mob.Test.tap(node, :tag)` — drive a tap by tag atom - `Mob.Test.find(node, "text")` — locate a widget by visible text diff --git a/lib/mob/test.ex b/lib/mob/test.ex index 5e357a2a..1f70e3e3 100644 --- a/lib/mob/test.ex +++ b/lib/mob/test.ex @@ -433,6 +433,13 @@ defmodule Mob.Test do A node whose `load_nif` failed reports `dist_rpc: true` with every probe `false`: it answered, and every NIF really is down. + Two side effects worth knowing. `:mob_nif` is `-on_load`, so calling this on + a node that has not loaded it triggers the code load and the NIF load — in a + booted Mob app it is always loaded already, so this is theory rather than + practice. And the device runs an interactive code server, so probing a module + it has not loaded causes it to load; that makes the answer reflect the code + path rather than the resident set. + An app built before `mob_nif:capabilities/0` existed cannot answer. Rather than guess from a table that would drift the same way, those report `:unknown` for each probe with `dist_rpc: true` — the honest answer, and one