Skip to content

iOS harness: stop tap_xy reporting phantom successes; add pixel sampling for colour - #80

Merged
GenericJam merged 5 commits into
masterfrom
fix/ios-test-harness-honesty
Aug 30, 2026
Merged

iOS harness: stop tap_xy reporting phantom successes; add pixel sampling for colour#80
GenericJam merged 5 commits into
masterfrom
fix/ios-test-harness-honesty

Conversation

@GenericJam

Copy link
Copy Markdown
Owner

The on-device test harness reported success for things that did not happen, and could not see colour at all. Both made agent-driven verification on iOS unreliable — an iOS renderer bug sat unverified in a downstream repo for weeks partly because of this.

1. tap_xy reported :ok when nothing happened

On a physical iPhone it returned :ok for every coordinate — including empty space — and nothing ever happened; mob_send_touch_phase returns YES for API availability, never delivery. On the simulator, tapping a Box with on_tap: returned :ok while the handler never ran, because SwiftUI gives a plain .onTapGesture no accessibility action.

Now the return value reflects whether a handler actually ran: a process-wide atomic counter is bumped by every send helper that routes a user event into the BEAM (mob_send_tap, mob_send_event, mob_send_change), sampled before injection and polled for 300 ms after.

ValueWhen
:okan event reached the BEAM within 300 ms
{:error, :no_view_at_point}hit-test found nothing (now on both platforms)
{:error, :no_effect}input accepted, no handler ran
{:error, :no_element_at_point}simulator: view present, no AX element

@doc now states the real per-platform capability, including that coordinate tapping does not work on a physical device and that the counter only sees Mob's own handlers (sidecar caveat). tap_xy also moved to ERL_NIF_DIRTY_JOB_IO_BOUND — it was already sleeping on a normal scheduler.

Device-verified: a theme chip (Box + on_tap) now returns {:error, :no_effect} where it previously returned a false :ok.

2. view_tree colour: measured, then abandoned as designed

Reading UIView.backgroundColor/UILabel.textColor gave colour for 2 of 443 nodes. Also walking CALayer/CAShapeLayer/CATextLayer reached 4 of 443. The new ui_paint_debug/0 census says why:

204x SwiftUI._UIInheritedView / CALayer all paint props 0
64x SwiftUI._UIInheritedView / SwiftUI.SDFLayer all paint props 0
14x _UIInheritedView / CGDrawingLayer all 0, contents=14

SwiftUI on iOS 26 paints via SDFLayer or rasterises into contents, exposing no readable paint property. Layer introspection is a dead end for SwiftUI content — recorded in an ADR rather than iterated on further. The colour fields stay (correct for UIKit chrome, root background, future Android) alongside a new :class field so a caller can tell which renderer drew a node and therefore why a colour is nil.

3. sample_color/2 — colour verification that works

sample_region/4 returns raw RGBA for a natively cropped region (small payload over distribution), reduced by a pure, unit-tested Elixir function to average, dominant, dominant_share and distinct — because a card with text on it is not one flat colour, and the share/distinct fields tell you whether to trust dominant.

Gated strictly#if !MOB_RELEASE, deliberately not the || defined(MOB_ENABLE_SCREENSHOT) its neighbour uses: arbitrary-rect pixel reads let a caller reconstruct the screen region by region, which would hand release builds the capability that opt-in exists to gate. Pinned by a test on the guard string.

Acceptance test, on device: two chips, background: :primary vs :surface_raised, under a glass theme.

Buildactiveinactive
without the glass fix0xFF20202E0xFF20202Ebug reproduced
with #780xFF4000AF0xFF101023fix confirmed

The regression that previously required hand pixel-diffing is now assert a.dominant != b.dominant.

Also reported, not fixed

scroll_info, scroll_to and element_frames are behind #if !MOB_RELEASE on iOS while Android registers them unconditionally, so an iOS release build raises nif_error(not_loaded) and kills the calling screen process. None of the three touches private API — the screenshot opt-in pattern would fit.

Verification

1011 tests passing, format/credo --strict/erlfmt/clang-format clean; ObjC compiled and run on an iPhone 17 simulator; non-vacuity of new tests proven by sabotage-and-restore.

🤖 Generated with Claude Code

@GenericJam

GenericJam commented Aug 30, 2026

Copy link
Copy Markdown
OwnerAuthor

Adversarial review + on-device verification complete. Verdict: approve the substance — rebase required before merge.

The branch forked at 0.7.20; master has 15 commits touching ios/mob_nif.m since. Merge-tree shows exactly one conflict hunk, in nif_tap_xy: master's side is MOB-99's retry-hardened atomic tap (mob_retry_main_thread_bool + find_a11y_at_point_in_current_windows); this branch still has the pre-MOB-99 two-dispatch_sync shape. Taking this side verbatim would regress MOB-99 — the counter/hit-test-first/await logic needs folding into MOB-99's atomic block. Everything else auto-merges.

The substance is NOT superseded: verified on current master that the device path still returns :ok off API availability and the sim path still returns :ok on accessibilityActivate with no effect check — the phantom-success bug this fixes is still live. Rebasing also fixes the id-addressed sampler path for free (reproduced the MOB-102 element_frames staleness on this branch on-device; rect-addressed sampling unaffected).

On-device (sim, 1011/1011 tests + all format/lint gates green on the branch): miss → {:error, :no_effect}; outside windows → {:error, :no_view_at_point}; real button → :ok with assigns actually changing; Box with on_tap → honest {:error, :no_effect} (previously phantom :ok). Pixel sampler round-trips sRGB token hexes bit-exact through P3/16-bit capture (Onyx 0xFF282424, Ivory 0xFFFFFAF4, share 1.0) — this can replace the downstream token-audit's 16-bit-unsupported bailout outright.

Two non-blocking notes for the rebase: (1) g_ui_event_seq is process-wide and scroll notifications bump it, so an unrelated event in the 300ms settle window can false-positive — fine for a serial harness, worth a doc sentence; (2) post-#94 the hard 'Box has no AX element' wording should be retested/softened (#94 added .isButton but no accessibilityAction, so the claim likely still holds). Nit: unchecked enif_alloc_binary returns in nif_sample_region/nif_ui_paint_debug.

GenericJamand others added 5 commits August 30, 2026 00:25
…to view_tree
tap_xy/2 returned ok whenever the platform input API accepted the event,
never when the app reacted. On the simulator accessibilityActivate
succeeds on a Box with on_tap while SwiftUI never runs the handler; on a
physical device the injected IOHID touch is accepted for every
coordinate and delivered for none. Both reported ok, so an agent driving
a device could not tell a working tap from a no-op.
Bump a process-wide counter from every send helper that routes a
user-originated event into the BEAM, sample it around the injection, and
poll for 300ms. ok now means the app demonstrably reacted; everything
else is a typed error (no_view_at_point / no_element_at_point /
no_effect). Hit-test up front on both paths, not just the device one.
tap_xy moves to a dirty IO scheduler since it can now block.
view_tree returned no colour at all, so a styling regression that
dropped every Box background was invisible to the one introspection API
meant to show what the device drew. Each node now carries bg_color and
text_color read back off UIView/CALayer as 0xAARRGGBB ints — the repo's
canonical colour representation (guides/theming.md). Documented the same
keys on the Android bridge contract, which no shipped MobBridge.kt
implements yet.
Fixes a latent bug the new tests surfaced: :json.decode maps JSON null to
:null, which normalize_view_tree passed straight through, so Android
labels compared unequal to nil and a null children list would have
crashed Enum.map.
Docs state the real per-platform capability instead of the aspirational
one, in Mob.Test, mob_nif.erl, CLAUDE.md and AGENTS.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds :mob_nif.ui_paint_debug/0, a census of which view/layer classes hold
readable paint, and a :class field on every view_tree node so a caller can
tell which renderer drew a node — and therefore why a colour is nil.
Extends colour extraction to the layer tree (CALayer.backgroundColor,
CAShapeLayer.fillColor, CATextLayer.foregroundColor). Measured on device
that takes it from 2 of 443 nodes to 4 of 443: SwiftUI on iOS 26 paints
through SDFLayer or rasterises into contents, so app content carries no
readable paint property at all. See the decision record — colour
verification needs screenshot sampling, not layer introspection.
view_tree colour is nil for virtually all SwiftUI content on iOS 26 (4 of
443 nodes when measured), so a styling regression like the glass theme that
discarded every Box background was only findable by pixel-diffing screenshots
by hand. Sampling pixels is the only reliable answer.
New NIF sample_region/4 (iOS) returns raw RGBA for a cropped region, with the
crop done inside the render so one element's pixels cross distribution instead
of a framebuffer. It shares screenshot/3's capture path via the extracted
mob_capture_window/mob_capture_image helpers.
Mob.Test.sample_color/2 addresses a region by element :id (through
element_frames) or an explicit rect, and reduce_rgba/3 is the pure reduction:
average, dominant, dominant_share, distinct, pixels — because a card with text
on it is not one flat colour and a bare mean of it is misleading either way.
sample_region stays strictly #if !MOB_RELEASE rather than joining screenshot's
MOB_ENABLE_SCREENSHOT opt-in: arbitrary-rect pixel reads reconstruct the screen
region by region, which would silently grant the capability that opt-in exists
to make deliberate. Pinned by a test.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Formatting only. These lines came in with the view_tree colour work earlier on
this branch (f1fb5e0, d04118c) and fail `xcrun clang-format --dry-run -Werror`,
which the pre-commit checklist runs. Kept out of the sampling commit so that
diff stays reviewable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…Box prose
- check enif_alloc_binary in nif_sample_region and nif_ui_paint_debug;
return {error, alloc_failed} instead of writing through a dead binary
- document that g_ui_event_seq is process-wide (scroll notifications bump
it too), so the 300ms settle check assumes a serial harness — an
unrelated Mob event inside the window reads as the tap's effect
- update the Box AX prose to post-#94 reality: accessibility_role "button"
makes Box a real AX element (.isButton) but adds no accessibilityAction,
so accessibilityActivate still doesn't fire on_tap (verified on-sim:
tap_xy on a role-button Box returns {error, no_effect})
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@GenericJam
GenericJamforce-pushed the fix/ios-test-harness-honesty branch from 39863be to 4a88f6fCompareAugust 30, 2026 07:02
@GenericJam

Copy link
Copy Markdown
OwnerAuthor

Rebased onto master (848eabd) with the MOB-99 fold done as planned: MOB-99's atomic retry skeleton kept verbatim (mob_retry_main_thread_bool, find+act in the same window, text-field responder walk), with the honesty logic folded in — seq snapshot before injection, hit-test-first {:error, :no_view_at_point}, post-activation settle wait → {:error, :no_effect}. Retry only re-runs find/activate, so no double-fire; settle runs once after a successful activation.

Re-verified on the rebased HEAD (4a88f6f): 1276 tests / 0 failures, format, credo --strict, warnings-as-errors, erlfmt, clang-format all clean. Device matrix re-run on a pool sim: honest miss/out-of-window errors, real button :ok with assigns advancing, 5 rapid taps all :ok (MOB-99 intact), tap-during-re-render :ok, sampler bit-exact (0xFF282424 / 0xFFFFAF4 wait — 0xFFFFFAF4, share 1.0), id-addressed sample_color stable across 3 forced re-renders (MOB-102 fix confirmed working with this branch).

Also included a review-fixes commit: enif_alloc_binary returns checked in nif_sample_region/nif_ui_paint_debug; process-wide-counter caveat documented beside the sidecar caveat (scroll events bump the counter — serial-harness assumption); and the post-#94 Box prose corrected to the device-verified truth: a role-button Box IS a findable AX element with .isButton but exposes no accessibilityAction, so activation is a verified no-op → honest {:error, :no_effect}.

Merging.

@GenericJam
GenericJam merged commit 6cf6d59 into masterAug 30, 2026
3 checks passed
GenericJam added a commit that referenced this pull request Aug 30, 2026
Written minutes before #80 merged, the honesty-contract and
evidence-matching sections claimed no pixel-sampling API existed and
leaned solely on state-change assertions. Now that tap_xy/3 reports
observed effect, both guides document the contract: :ok only when an
event reached the BEAM within 300ms, else {:error, :no_view_at_point |
:no_element_at_point | :no_effect} — plus the platform limits that make
:no_effect legitimate (SwiftUI on_tap containers, physical-device
injection) and the serial-harness assumption the 300ms window shares
with state-change checks.
Evidence matching now splits exact-color decisions (sample_color/2:
real pixels, dominant/average as 0xAARRGGBB, iOS debug-build only) from
holistic visual parity (screenshots with tolerance). testing.md gains
matching sections.
No change needed for #77: push_notifications.md already described
tap-to-open from a killed app, which that fix made true.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GenericJam added a commit that referenced this pull request Aug 30, 2026
…eature coverage (#105)
* docs(navigation): multi-stack state, honest tab/drawer claims, back semantics
navigation.md still described tab_bar/drawer as rendering native chrome
(UITabBarController / NavigationBar) — since 0.7.33 the runtime backs the
declaration with real per-stack state but draws no chrome; switching is
programmatic. New 'Tabs and multi-stack state' section documents lazy
materialization, parking, independent histories, back-at-secondary-root,
the orphan stack, and the MOB-115/116/117 gaps. Directional-reset docs
gain the ArgumentError validation and transition-survives-coalescing
behavior (#100/#103).
screen_lifecycle.md gains crash/restart semantics (per-screen isolation,
restart cap, re-mount + load_state), per-screen self() and message
delivery, terminate/2 reality (pop stops the leaving screen only), and
multi-stack system-back.
Mob.App.tab_bar/1 and drawer/1 docstrings no longer claim chrome that is
not drawn.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(testing): document Mob.ScreenCase and settle/2; remove nonexistent screen_pid/1
The testing guide never mentioned Mob.ScreenCase (#44), the blessed
in-BEAM unit-test path — it now leads the guide. The old sync-point
advice referenced Mob.Test.screen_pid/1, which does not exist, and
:sys.get_state on :mob_screen, which stopped being sufficient when
rendering moved to Mob.Sender (MOB-110) and :mob_screen became the
navigation owner (MOB-112); both are replaced with Mob.Test.settle/2
and an explanation of the three processes it drains. Unit-test examples
updated for the process model (dispatch is synchronous; get_socket is
the natural sync point after send).
Mob.Test's moduledoc claimed tap/2 goes through handle_event/3; it
sends {:tap, tag} to handle_info/2 like a real native tap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(components,theming): Sheet section, handle-pool limits, font tokens
components.md had no coverage of Mob.UI.sheet/2 (0.7.29) or intrinsic
content detents (0.7.32) beyond the surface-matrix row — new ':sheet'
section documents presence-is-presentation, detents including
[:content] / [{:content, max_height: n}], exactly-once {:dismiss, tag},
and the iOS scrim limitation. New 'Handle limits' section covers the
256-handle tap pool and the 256-slot native component pool with
{:error, :component_slots_exhausted} (0.7.28 behavior). :text gains the
font prop (named font tokens, 0.7.25).
theming.md never mentioned fonts — adds the fonts:/font_fallback: token
type with a pointer to Styling → Custom fonts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: fix tap examples to handle_info/2 and stale module/function names
README and several guides showed UI taps handled by
handle_event("tap", %{"tag" => ...}) — a real tap delivers
{:tap, tag} to handle_info/2, so those example screens would never
respond on a device. README's diagram and testing snippet updated for
the per-screen process model and Mob.ScreenCase. getting_started
referenced Mob.Nav.push/2, which does not exist (Mob.Socket.push_screen
is the API). event_audit's list-select re-emitter is Mob.Screen.Server
since MOB-113.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(agentic): split into single-agent and agent-team halves; new practices
Part 1 (Working with one agent) keeps the existing content in order and
adds: verify effects not exit codes (assert the app answers after a
deploy), the honesty contract (success = a handler ran — assert on state
change after a tap, settle-window caveat), match the evidence to the
question (frames for layout, screenshots-with-tolerance for appearance,
recordings for motion), lifecycle-event simulation recipes (simctl push
.apns, adb broadcast / cmd notification post), and environment
discipline (complete .tool-versions incl. zig/JDK, the
MOB_DIR/MOB_DEV_DIR/MOB_NEW_DIR override chain).
Part 2 (Working with agent teams) is new: one driver per device with
lease discipline (humans outrank agents), unique node names per session
(mob.connect --name), per-task git worktrees, the mob.push/mob.watch
fan-out hazard (they reach every live node, no device scoping — fleets
deploy per device or push over their own dist connection), and durable
artifacts as the handoff medium between context windows.
The standard agent loop gains a settle/2 step before native-side reads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(mix): group new modules in hexdocs sidebar
Mob.Router joins Navigation; Mob.Screen.Server, Mob.Listener and
Mob.Sender get a Runtime Processes group; Mob.ScreenCase joins
Testing & Debugging. All five were shipping ungrouped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: document #80's honest tap returns and pixel sampling
Written minutes before #80 merged, the honesty-contract and
evidence-matching sections claimed no pixel-sampling API existed and
leaned solely on state-change assertions. Now that tap_xy/3 reports
observed effect, both guides document the contract: :ok only when an
event reached the BEAM within 300ms, else {:error, :no_view_at_point |
:no_element_at_point | :no_effect} — plus the platform limits that make
:no_effect legitimate (SwiftUI on_tap containers, physical-device
injection) and the serial-harness assumption the 300ms window shares
with state-change checks.
Evidence matching now splits exact-color decisions (sample_color/2:
real pixels, dominant/average as 0xAARRGGBB, iOS debug-build only) from
holistic visual parity (screenshots with tolerance). testing.md gains
matching sections.
No change needed for #77: push_notifications.md already described
tap-to-open from a killed app, which that fix made true.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
GenericJam added a commit that referenced this pull request Sep 1, 2026
Written minutes before #80 merged, the honesty-contract and
evidence-matching sections claimed no pixel-sampling API existed and
leaned solely on state-change assertions. Now that tap_xy/3 reports
observed effect, both guides document the contract: :ok only when an
event reached the BEAM within 300ms, else {:error, :no_view_at_point |
:no_element_at_point | :no_effect} — plus the platform limits that make
:no_effect legitimate (SwiftUI on_tap containers, physical-device
injection) and the serial-harness assumption the 300ms window shares
with state-change checks.
Evidence matching now splits exact-color decisions (sample_color/2:
real pixels, dominant/average as 0xAARRGGBB, iOS debug-build only) from
holistic visual parity (screenshots with tolerance). testing.md gains
matching sections.
No change needed for #77: push_notifications.md already described
tap-to-open from a killed app, which that fix made true.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@GenericJam