iOS harness: stop tap_xy reporting phantom successes; add pixel sampling for colour - #80
Conversation
Adversarial review + on-device verification complete. Verdict: approve the substance — rebase required before merge. The branch forked at 0.7.20; master has 15 commits touching The substance is NOT superseded: verified on current master that the device path still returns On-device (sim, 1011/1011 tests + all format/lint gates green on the branch): miss → Two non-blocking notes for the rebase: (1) |
…to view_tree tap_xy/2 returned ok whenever the platform input API accepted the event, never when the app reacted. On the simulator accessibilityActivate succeeds on a Box with on_tap while SwiftUI never runs the handler; on a physical device the injected IOHID touch is accepted for every coordinate and delivered for none. Both reported ok, so an agent driving a device could not tell a working tap from a no-op. Bump a process-wide counter from every send helper that routes a user-originated event into the BEAM, sample it around the injection, and poll for 300ms. ok now means the app demonstrably reacted; everything else is a typed error (no_view_at_point / no_element_at_point / no_effect). Hit-test up front on both paths, not just the device one. tap_xy moves to a dirty IO scheduler since it can now block. view_tree returned no colour at all, so a styling regression that dropped every Box background was invisible to the one introspection API meant to show what the device drew. Each node now carries bg_color and text_color read back off UIView/CALayer as 0xAARRGGBB ints — the repo's canonical colour representation (guides/theming.md). Documented the same keys on the Android bridge contract, which no shipped MobBridge.kt implements yet. Fixes a latent bug the new tests surfaced: :json.decode maps JSON null to :null, which normalize_view_tree passed straight through, so Android labels compared unequal to nil and a null children list would have crashed Enum.map. Docs state the real per-platform capability instead of the aspirational one, in Mob.Test, mob_nif.erl, CLAUDE.md and AGENTS.md. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds :mob_nif.ui_paint_debug/0, a census of which view/layer classes hold readable paint, and a :class field on every view_tree node so a caller can tell which renderer drew a node — and therefore why a colour is nil. Extends colour extraction to the layer tree (CALayer.backgroundColor, CAShapeLayer.fillColor, CATextLayer.foregroundColor). Measured on device that takes it from 2 of 443 nodes to 4 of 443: SwiftUI on iOS 26 paints through SDFLayer or rasterises into contents, so app content carries no readable paint property at all. See the decision record — colour verification needs screenshot sampling, not layer introspection.
view_tree colour is nil for virtually all SwiftUI content on iOS 26 (4 of 443 nodes when measured), so a styling regression like the glass theme that discarded every Box background was only findable by pixel-diffing screenshots by hand. Sampling pixels is the only reliable answer. New NIF sample_region/4 (iOS) returns raw RGBA for a cropped region, with the crop done inside the render so one element's pixels cross distribution instead of a framebuffer. It shares screenshot/3's capture path via the extracted mob_capture_window/mob_capture_image helpers. Mob.Test.sample_color/2 addresses a region by element :id (through element_frames) or an explicit rect, and reduce_rgba/3 is the pure reduction: average, dominant, dominant_share, distinct, pixels — because a card with text on it is not one flat colour and a bare mean of it is misleading either way. sample_region stays strictly #if !MOB_RELEASE rather than joining screenshot's MOB_ENABLE_SCREENSHOT opt-in: arbitrary-rect pixel reads reconstruct the screen region by region, which would silently grant the capability that opt-in exists to make deliberate. Pinned by a test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Formatting only. These lines came in with the view_tree colour work earlier on this branch (f1fb5e0, d04118c) and fail `xcrun clang-format --dry-run -Werror`, which the pre-commit checklist runs. Kept out of the sampling commit so that diff stays reviewable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…Box prose
- check enif_alloc_binary in nif_sample_region and nif_ui_paint_debug;
return {error, alloc_failed} instead of writing through a dead binary
- document that g_ui_event_seq is process-wide (scroll notifications bump
it too), so the 300ms settle check assumes a serial harness — an
unrelated Mob event inside the window reads as the tap's effect
- update the Box AX prose to post-#94 reality: accessibility_role "button"
makes Box a real AX element (.isButton) but adds no accessibilityAction,
so accessibilityActivate still doesn't fire on_tap (verified on-sim:
tap_xy on a role-button Box returns {error, no_effect})
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>39863be to
4a88f6fCompareGenericJam
commented
Aug 30, 2026
Rebased onto master (848eabd) with the MOB-99 fold done as planned: MOB-99's atomic retry skeleton kept verbatim (mob_retry_main_thread_bool, find+act in the same window, text-field responder walk), with the honesty logic folded in — seq snapshot before injection, hit-test-first {:error, :no_view_at_point}, post-activation settle wait → {:error, :no_effect}. Retry only re-runs find/activate, so no double-fire; settle runs once after a successful activation. Re-verified on the rebased HEAD (4a88f6f): 1276 tests / 0 failures, format, credo --strict, warnings-as-errors, erlfmt, clang-format all clean. Device matrix re-run on a pool sim: honest miss/out-of-window errors, real button :ok with assigns advancing, 5 rapid taps all :ok (MOB-99 intact), tap-during-re-render :ok, sampler bit-exact (0xFF282424 / 0xFFFFAF4 wait — 0xFFFFFAF4, share 1.0), id-addressed sample_color stable across 3 forced re-renders (MOB-102 fix confirmed working with this branch). Also included a review-fixes commit: enif_alloc_binary returns checked in nif_sample_region/nif_ui_paint_debug; process-wide-counter caveat documented beside the sidecar caveat (scroll events bump the counter — serial-harness assumption); and the post-#94 Box prose corrected to the device-verified truth: a role-button Box IS a findable AX element with .isButton but exposes no accessibilityAction, so activation is a verified no-op → honest {:error, :no_effect}. Merging. |
Uh oh!
There was an error while loading. Please reload this page.
Written minutes before #80 merged, the honesty-contract and evidence-matching sections claimed no pixel-sampling API existed and leaned solely on state-change assertions. Now that tap_xy/3 reports observed effect, both guides document the contract: :ok only when an event reached the BEAM within 300ms, else {:error, :no_view_at_point | :no_element_at_point | :no_effect} — plus the platform limits that make :no_effect legitimate (SwiftUI on_tap containers, physical-device injection) and the serial-harness assumption the 300ms window shares with state-change checks. Evidence matching now splits exact-color decisions (sample_color/2: real pixels, dominant/average as 0xAARRGGBB, iOS debug-build only) from holistic visual parity (screenshots with tolerance). testing.md gains matching sections. No change needed for #77: push_notifications.md already described tap-to-open from a killed app, which that fix made true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…eature coverage (#105) * docs(navigation): multi-stack state, honest tab/drawer claims, back semantics navigation.md still described tab_bar/drawer as rendering native chrome (UITabBarController / NavigationBar) — since 0.7.33 the runtime backs the declaration with real per-stack state but draws no chrome; switching is programmatic. New 'Tabs and multi-stack state' section documents lazy materialization, parking, independent histories, back-at-secondary-root, the orphan stack, and the MOB-115/116/117 gaps. Directional-reset docs gain the ArgumentError validation and transition-survives-coalescing behavior (#100/#103). screen_lifecycle.md gains crash/restart semantics (per-screen isolation, restart cap, re-mount + load_state), per-screen self() and message delivery, terminate/2 reality (pop stops the leaving screen only), and multi-stack system-back. Mob.App.tab_bar/1 and drawer/1 docstrings no longer claim chrome that is not drawn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(testing): document Mob.ScreenCase and settle/2; remove nonexistent screen_pid/1 The testing guide never mentioned Mob.ScreenCase (#44), the blessed in-BEAM unit-test path — it now leads the guide. The old sync-point advice referenced Mob.Test.screen_pid/1, which does not exist, and :sys.get_state on :mob_screen, which stopped being sufficient when rendering moved to Mob.Sender (MOB-110) and :mob_screen became the navigation owner (MOB-112); both are replaced with Mob.Test.settle/2 and an explanation of the three processes it drains. Unit-test examples updated for the process model (dispatch is synchronous; get_socket is the natural sync point after send). Mob.Test's moduledoc claimed tap/2 goes through handle_event/3; it sends {:tap, tag} to handle_info/2 like a real native tap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(components,theming): Sheet section, handle-pool limits, font tokens components.md had no coverage of Mob.UI.sheet/2 (0.7.29) or intrinsic content detents (0.7.32) beyond the surface-matrix row — new ':sheet' section documents presence-is-presentation, detents including [:content] / [{:content, max_height: n}], exactly-once {:dismiss, tag}, and the iOS scrim limitation. New 'Handle limits' section covers the 256-handle tap pool and the 256-slot native component pool with {:error, :component_slots_exhausted} (0.7.28 behavior). :text gains the font prop (named font tokens, 0.7.25). theming.md never mentioned fonts — adds the fonts:/font_fallback: token type with a pointer to Styling → Custom fonts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: fix tap examples to handle_info/2 and stale module/function names README and several guides showed UI taps handled by handle_event("tap", %{"tag" => ...}) — a real tap delivers {:tap, tag} to handle_info/2, so those example screens would never respond on a device. README's diagram and testing snippet updated for the per-screen process model and Mob.ScreenCase. getting_started referenced Mob.Nav.push/2, which does not exist (Mob.Socket.push_screen is the API). event_audit's list-select re-emitter is Mob.Screen.Server since MOB-113. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(agentic): split into single-agent and agent-team halves; new practices Part 1 (Working with one agent) keeps the existing content in order and adds: verify effects not exit codes (assert the app answers after a deploy), the honesty contract (success = a handler ran — assert on state change after a tap, settle-window caveat), match the evidence to the question (frames for layout, screenshots-with-tolerance for appearance, recordings for motion), lifecycle-event simulation recipes (simctl push .apns, adb broadcast / cmd notification post), and environment discipline (complete .tool-versions incl. zig/JDK, the MOB_DIR/MOB_DEV_DIR/MOB_NEW_DIR override chain). Part 2 (Working with agent teams) is new: one driver per device with lease discipline (humans outrank agents), unique node names per session (mob.connect --name), per-task git worktrees, the mob.push/mob.watch fan-out hazard (they reach every live node, no device scoping — fleets deploy per device or push over their own dist connection), and durable artifacts as the handoff medium between context windows. The standard agent loop gains a settle/2 step before native-side reads. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs(mix): group new modules in hexdocs sidebar Mob.Router joins Navigation; Mob.Screen.Server, Mob.Listener and Mob.Sender get a Runtime Processes group; Mob.ScreenCase joins Testing & Debugging. All five were shipping ungrouped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: document #80's honest tap returns and pixel sampling Written minutes before #80 merged, the honesty-contract and evidence-matching sections claimed no pixel-sampling API existed and leaned solely on state-change assertions. Now that tap_xy/3 reports observed effect, both guides document the contract: :ok only when an event reached the BEAM within 300ms, else {:error, :no_view_at_point | :no_element_at_point | :no_effect} — plus the platform limits that make :no_effect legitimate (SwiftUI on_tap containers, physical-device injection) and the serial-harness assumption the 300ms window shares with state-change checks. Evidence matching now splits exact-color decisions (sample_color/2: real pixels, dominant/average as 0xAARRGGBB, iOS debug-build only) from holistic visual parity (screenshots with tolerance). testing.md gains matching sections. No change needed for #77: push_notifications.md already described tap-to-open from a killed app, which that fix made true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Written minutes before #80 merged, the honesty-contract and evidence-matching sections claimed no pixel-sampling API existed and leaned solely on state-change assertions. Now that tap_xy/3 reports observed effect, both guides document the contract: :ok only when an event reached the BEAM within 300ms, else {:error, :no_view_at_point | :no_element_at_point | :no_effect} — plus the platform limits that make :no_effect legitimate (SwiftUI on_tap containers, physical-device injection) and the serial-harness assumption the 300ms window shares with state-change checks. Evidence matching now splits exact-color decisions (sample_color/2: real pixels, dominant/average as 0xAARRGGBB, iOS debug-build only) from holistic visual parity (screenshots with tolerance). testing.md gains matching sections. No change needed for #77: push_notifications.md already described tap-to-open from a killed app, which that fix made true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The on-device test harness reported success for things that did not happen, and could not see colour at all. Both made agent-driven verification on iOS unreliable — an iOS renderer bug sat unverified in a downstream repo for weeks partly because of this.
1.
tap_xyreported:okwhen nothing happenedOn a physical iPhone it returned
:okfor every coordinate — including empty space — and nothing ever happened;mob_send_touch_phasereturnsYESfor API availability, never delivery. On the simulator, tapping a Box withon_tap:returned:okwhile the handler never ran, because SwiftUI gives a plain.onTapGestureno accessibility action.Now the return value reflects whether a handler actually ran: a process-wide atomic counter is bumped by every send helper that routes a user event into the BEAM (
mob_send_tap,mob_send_event,mob_send_change), sampled before injection and polled for 300 ms after.:ok{:error, :no_view_at_point}{:error, :no_effect}{:error, :no_element_at_point}@docnow states the real per-platform capability, including that coordinate tapping does not work on a physical device and that the counter only sees Mob's own handlers (sidecar caveat).tap_xyalso moved toERL_NIF_DIRTY_JOB_IO_BOUND— it was already sleeping on a normal scheduler.Device-verified: a theme chip (Box +
on_tap) now returns{:error, :no_effect}where it previously returned a false:ok.2.
view_treecolour: measured, then abandoned as designedReading
UIView.backgroundColor/UILabel.textColorgave colour for 2 of 443 nodes. Also walkingCALayer/CAShapeLayer/CATextLayerreached 4 of 443. The newui_paint_debug/0census says why:SwiftUI on iOS 26 paints via
SDFLayeror rasterises intocontents, exposing no readable paint property. Layer introspection is a dead end for SwiftUI content — recorded in an ADR rather than iterated on further. The colour fields stay (correct for UIKit chrome, root background, future Android) alongside a new:classfield so a caller can tell which renderer drew a node and therefore why a colour isnil.3.
sample_color/2— colour verification that workssample_region/4returns raw RGBA for a natively cropped region (small payload over distribution), reduced by a pure, unit-tested Elixir function toaverage,dominant,dominant_shareanddistinct— because a card with text on it is not one flat colour, and the share/distinct fields tell you whether to trustdominant.Gated strictly
#if !MOB_RELEASE, deliberately not the|| defined(MOB_ENABLE_SCREENSHOT)its neighbour uses: arbitrary-rect pixel reads let a caller reconstruct the screen region by region, which would hand release builds the capability that opt-in exists to gate. Pinned by a test on the guard string.Acceptance test, on device: two chips,
background: :primaryvs:surface_raised, under a glass theme.0xFF20202E0xFF20202E0xFF4000AF0xFF101023The regression that previously required hand pixel-diffing is now
assert a.dominant != b.dominant.Also reported, not fixed
scroll_info,scroll_toandelement_framesare behind#if !MOB_RELEASEon iOS while Android registers them unconditionally, so an iOS release build raisesnif_error(not_loaded)and kills the calling screen process. None of the three touches private API — thescreenshotopt-in pattern would fit.Verification
1011 tests passing,
format/credo --strict/erlfmt/clang-formatclean; ObjC compiled and run on an iPhone 17 simulator; non-vacuity of new tests proven by sabotage-and-restore.🤖 Generated with Claude Code