Skip to content

feat(ui): add intrinsic sheets and box semantics - #94

Merged
GenericJam merged 3 commits into
masterfrom
feat/intrinsic-sheet-detents
Aug 28, 2026
Merged

feat(ui): add intrinsic sheets and box semantics#94
GenericJam merged 3 commits into
masterfrom
feat/intrinsic-sheet-detents

Conversation

@GenericJam

@GenericJam GenericJam commented Aug 27, 2026

Copy link
Copy Markdown
Owner

Summary

  • add intrinsic-content Sheet detents, including an optional maximum height
  • validate and encode detents at the renderer boundary so raw/sigil nodes cannot bypass the contract
  • measure, cap, re-clamp, and internally scroll intrinsic Sheet content on iOS
  • expose generic Box accessibility labels, button roles, and disabled semantics
  • include Sheet in both packaged platform tag manifests

Why this belongs in Mob

These are generic renderer capabilities used by composed UI libraries. A plugin can style and compose Mob.UI.sheet/2, but it cannot change how the core Sheet measures arbitrary Mob children or make the core Box renderer honor accessibility props without replacing those primitives with app-specific native leaves.

The coordinated generated-Android implementation is GenericJam/mob_new#39.

Verification

  • mix format --check-formatted
  • mix credo --strict
  • mix compile --warnings-as-errors
  • mix test --exclude macos_only --exclude requires_zig — 1,119 passed, 38 excluded
  • git diff --check origin/master..HEAD

This PR intentionally does not bump or publish a package version. Release bookkeeping follows after upstream review.

Eight findings from an adversarial pass on 894bf5d, plus two of my own.

iOS Box — three interacting fixes:

- .disabled(node.disabled) replaces .allowsHitTesting(!node.disabled).
  allowsHitTesting(false) makes a view TRANSPARENT to touches, so a disabled
  box used as a blocking overlay or dimmed backdrop passed taps straight
  through to the content it was meant to be shielding. .disabled blocks
  interaction in the subtree while still consuming the touch.

- Stop branching on `disabled`. `ifLet` is @ViewBuilder if/else, i.e.
  _ConditionalContent, so flipping the branch hands SwiftUI a structurally
  different view and tears the subtree down. `disabled` is routinely toggled:
  a box wrapping a TextField lost its in-flight text and focus, and one
  wrapping a Sheet re-presented it. Tap wiring now branches on handler
  presence only, with the disabled check inside the closure, and both traits
  are unconditional OptionSet modifiers.

- Collapse to one accessibility element for a label OR an explicit button
  role, not just a label. Traits added without collapsing land on every
  descendant, so a role-only box announced each nested Text as its own button.

iOS Sheet:

- Content detents no longer flash a 1pt hairline on presentation. Content can
  only be measured once the sheet is up, so the first detent was computed from
  sentinels (intrinsic 1, environment default 1) and resolved to .height(1) on
  EVERY presentation. Measurement is now Optional and unmeasured presents at
  .medium; the environment default 0 means "root geometry unknown" and no
  longer doubles as a clamp.

- The detent accounts for the sheet's bottom safe-area inset. .height() is
  TOTAL sheet height while the content region is inset by the home indicator,
  so an intrinsic sheet was ~34pt short on notched devices — last rows under
  the indicator, and scrollable when the content was meant to fit exactly.
  The preference now carries height and inset together.

- The 0.9 ceiling is a named constant with its rationale.

Elixir:

- Restored per-rule validation messages that echo the offending value.
  [:medium, :medium] now says "must not contain duplicates, got: ..." instead
  of restating the whole grammar, and a malformed content map gets its own
  message rather than falling through to the built-in-subset branch.

- @external_resource on both tag manifests in Mob.Sigil and Mob.ScreenCase.
  They are read into module attributes at compile time, so adding Sheet
  recompiled nothing for anyone with a warm _build — every path-dep consumer,
  which is the standard mob dev setup — leaving the sigil rejecting <Sheet>
  until a manual mix clean.

- Documented the two content-detent limitations the implementation implies: a
  scrollable child expands rather than scrolling independently, and invalid
  detents now raise at the renderer boundary instead of degrading.

Test quality: the native grep tests broke on four separate behaviour-preserving
refactors here (naming a constant, unwrapping an Optional, adding the inset,
making traits unconditional). They pinned source text, not behaviour. Retargeted
at the mechanisms — the unmeasured fallback, the inset arithmetic, the
non-branching traits, .disabled over .allowsHitTesting — so they break when the
behaviour regresses rather than when the code is tidied.

1119 tests, format, credo --strict, clang-format, swiftlint (only the
pre-existing force_cast), and clang -fsyntax-only on mob_nif.m all clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@GenericJam

Copy link
Copy Markdown
Owner Author

Adversarial review addressed — 34866b5

Ran a high-effort adversarial pass over 894bf5d. Eight findings, plus two of my own. All fixed on this branch.

iOS Box — three interacting fixes

.allowsHitTesting(false) let taps fall through a disabled box. It makes a view transparent to touches rather than blocking them, so a Box with disabled: true used as a blocking overlay or dimmed backdrop passed taps straight through to the content it was meant to shield. Now .disabled(node.disabled), which blocks interaction in the subtree while still consuming the touch.

Toggling disabled destroyed the subtree's SwiftUI identity. ifLet is @ViewBuilder if/else — _ConditionalContent — so flipping the branch hands SwiftUI a structurally different view and tears the subtree down. A Box wrapping a TextField lost in-flight text and focus; one wrapping a Sheet re-presented it. Tap wiring now branches on handler presence only, with the disabled check inside the closure, and both traits are unconditional OptionSet modifiers (? .isButton : []).

Traits leaked to descendants without a label. .accessibilityAddTraits is only scoped when .accessibilityElement(children: .ignore) ran, which only happened inside the label branch — so a role-only box announced every nested Text as its own button. Collapsing now keys on label or explicit button role.

iOS Sheet

Content detents flashed a 1pt hairline on every presentation. Content can only be measured after the sheet is up, so the first detent came from sentinels (intrinsicContentHeight = 1, environment default 1) and resolved to .height(1) — on every MobSheetView, not just the first. Measurement is now Optional; unmeasured presents at .medium and switches to the exact height when the measurement lands. The environment default is 0 meaning "root geometry unknown", so it no longer doubles as a clamp.

The detent ignored the bottom safe-area inset. .height() sets total sheet height while the content region is inset by the home indicator, so an intrinsic sheet was ~34pt short on notched devices — last rows under the indicator, and scrollable when the content was meant to fit exactly. The preference now carries height and inset together.

The 0.9 ceiling is a named constant with its rationale.

Elixir

Validation messages regressed to one generic string with no got:. Restored per-rule messages — [:medium, :medium] says "must not contain duplicates, got: …" — plus a dedicated message for malformed %{type: :content} maps, which previously fell through to the built-in-subset branch and misdescribed the problem.

@external_resource added to both tag manifests in Mob.Sigil and Mob.ScreenCase. They're read into module attributes at compile time, so adding Sheet recompiled nothing for anyone with a warm _build — every path-dep consumer, i.e. the standard mob dev setup — leaving the sigil rejecting <Sheet> until a manual mix clean. This change is the first to actually trip it.

Documented two limitations the implementation implies: a scrollable child (scroll, lazy_list) reports full content height under .fixedSize, so it expands rather than scrolling independently; and invalid detents now raise at the renderer boundary rather than degrading to [:medium, :large].

On the native tests

They broke on four separate behaviour-preserving refactors here — naming a constant, unwrapping an Optional, adding the inset, making traits unconditional. They pinned source text, not behaviour. Retargeted at the mechanisms (the unmeasured fallback, the inset arithmetic, the non-branching traits, .disabled over .allowsHitTesting) so they break when behaviour regresses rather than when the code is tidied.

Verification

1119 tests, mix format, credo --strict, clang-format, swiftlint (only the pre-existing force_cast), and clang -fsyntax-only on mob_nif.m — all clean. Device verification of the sheet still to come.

Device verification found what no Elixir-side check could: PR #94 as submitted
DID NOT BUILD for iOS. Two separate compile errors.

1. `AccessibilityTraits` has no member `.isNotEnabled`. The original
   `.accessibilityAddTraits(.isNotEnabled)` is not a real API — confirmed zero
   occurrences in the iOS 26.5 SwiftUI interface. Disabled state is not a trait
   you add; it is what `.disabled(true)` publishes to VoiceOver, which this
   code already does. Removed.

2. "The compiler is unable to type-check this expression in reasonable time."
   The accessibility and tap modifiers pushed MobBox's chain past the Swift
   type-inference budget. Extracted into a MobBoxSemantics ViewModifier — the
   same treatment MobBox itself already got from MobNodeView for the same
   reason.

Worth dwelling on how #1 survived review: the grep test asserted
`.accessibilityAddTraits(.isNotEnabled)` appears in the source, and it did.
The string was present and the build was broken. `mix test`, `credo`,
`mix format`, `clang-format`, `swiftlint`, and `clang -fsyntax-only` on
mob_nif.m all pass on code that cannot compile, because none of them type-check
Swift. Only a native build does. The test now refutes the call form.

Device results (iPhone 17 Pro simulator, 874pt tall — chosen over the attached
iPhone SE because the SE has no home indicator and so cannot exercise the
safe-area fix at all):

- short [:content], 3 rows: hugs content, rows at y=716..784. Sheet is sized to
  content, parent visible behind.
- tall, max_height: 320, 40 rows: sheet top ~y=541, i.e. capped at ~320pt;
  content extends to y=1864 and scrolls internally.
- tall [:content], 40 rows: sheet top ~y=147, capped near the 0.9 root ceiling
  (786pt); scrolls internally.
- exact fit, 8 rows: all 8 visible, last row bottom clears the home indicator
  rather than sitting under it — the safe-area inset fix, confirmed visually.
- Sampling element_frames every 120ms from presentation showed a stable layout
  from the first sample; no degenerate frame. The 1pt case is structurally gone
  now that measurement is Optional rather than a numeric sentinel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@GenericJam
GenericJam merged commit e0b2435 into master Aug 28, 2026
4 checks passed
@GenericJam
GenericJam deleted the feat/intrinsic-sheet-detents branch August 28, 2026 02:01
GenericJam added a commit that referenced this pull request Aug 28, 2026
…eadlock fix

Carries three merged changes since 0.7.31:

- #94 intrinsic Sheet detents ([:content] / [{:content, max_height: n}]) with
  canonical validation through Mob.UI.normalize_sheet_detents!/1 re-applied at
  the renderer boundary, iOS content measurement with capping, internal
  scrolling and geometry re-clamping, composite Box accessibility (label,
  explicit button role, disabled semantics), and Sheet in both platform tag
  manifests.
- #74 the iOS launch deadlock behind an App Store rejection — safe_area()'s
  unbounded dispatch_sync on the boot path could block the BEAM thread forever
  so the app never finished launching.
- #78 iOS glass tint from the node background, and left-aligned fill_width rows.

The changelog records two known limitations rather than leaving them to be
discovered: a scrollable child inside a content detent expands instead of
scrolling independently (it reports intrinsic height), and a content sheet
presents at :medium for one frame before resizing, since content height is
only knowable after presentation.

Preflight: mix format, credo --strict, compile --warnings-as-errors, 1119
tests, git diff --check, and the native frame-registry harness all clean.
The Sheet work was additionally adversarially reviewed and device-verified on
an iPhone 17 Pro simulator (hug, cap, internal scroll, safe-area clearance)
and the Box/Android half on a physical Moto G.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GenericJam added a commit that referenced this pull request Aug 30, 2026
…Box prose

- check enif_alloc_binary in nif_sample_region and nif_ui_paint_debug;
  return {error, alloc_failed} instead of writing through a dead binary
- document that g_ui_event_seq is process-wide (scroll notifications bump
  it too), so the 300ms settle check assumes a serial harness — an
  unrelated Mob event inside the window reads as the tap's effect
- update the Box AX prose to post-#94 reality: accessibility_role "button"
  makes Box a real AX element (.isButton) but adds no accessibilityAction,
  so accessibilityActivate still doesn't fire on_tap (verified on-sim:
  tap_xy on a role-button Box returns {error, no_effect})

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GenericJam added a commit that referenced this pull request Aug 30, 2026
…ing for colour (#80)

* iOS harness: make tap_xy report observed effect; add painted colours to view_tree

tap_xy/2 returned ok whenever the platform input API accepted the event,
never when the app reacted. On the simulator accessibilityActivate
succeeds on a Box with on_tap while SwiftUI never runs the handler; on a
physical device the injected IOHID touch is accepted for every
coordinate and delivered for none. Both reported ok, so an agent driving
a device could not tell a working tap from a no-op.

Bump a process-wide counter from every send helper that routes a
user-originated event into the BEAM, sample it around the injection, and
poll for 300ms. ok now means the app demonstrably reacted; everything
else is a typed error (no_view_at_point / no_element_at_point /
no_effect). Hit-test up front on both paths, not just the device one.
tap_xy moves to a dirty IO scheduler since it can now block.

view_tree returned no colour at all, so a styling regression that
dropped every Box background was invisible to the one introspection API
meant to show what the device drew. Each node now carries bg_color and
text_color read back off UIView/CALayer as 0xAARRGGBB ints — the repo's
canonical colour representation (guides/theming.md). Documented the same
keys on the Android bridge contract, which no shipped MobBridge.kt
implements yet.

Fixes a latent bug the new tests surfaced: :json.decode maps JSON null to
:null, which normalize_view_tree passed straight through, so Android
labels compared unequal to nil and a null children list would have
crashed Enum.map.

Docs state the real per-platform capability instead of the aspirational
one, in Mob.Test, mob_nif.erl, CLAUDE.md and AGENTS.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* harness: census where paint lives; keep view_tree colour best-effort

Adds :mob_nif.ui_paint_debug/0, a census of which view/layer classes hold
readable paint, and a :class field on every view_tree node so a caller can
tell which renderer drew a node — and therefore why a colour is nil.

Extends colour extraction to the layer tree (CALayer.backgroundColor,
CAShapeLayer.fillColor, CATextLayer.foregroundColor). Measured on device
that takes it from 2 of 443 nodes to 4 of 443: SwiftUI on iOS 26 paints
through SDFLayer or rasterises into contents, so app content carries no
readable paint property at all. See the decision record — colour
verification needs screenshot sampling, not layer introspection.

* harness: sample real pixels to verify drawn colour

view_tree colour is nil for virtually all SwiftUI content on iOS 26 (4 of
443 nodes when measured), so a styling regression like the glass theme that
discarded every Box background was only findable by pixel-diffing screenshots
by hand. Sampling pixels is the only reliable answer.

New NIF sample_region/4 (iOS) returns raw RGBA for a cropped region, with the
crop done inside the render so one element's pixels cross distribution instead
of a framebuffer. It shares screenshot/3's capture path via the extracted
mob_capture_window/mob_capture_image helpers.

Mob.Test.sample_color/2 addresses a region by element :id (through
element_frames) or an explicit rect, and reduce_rgba/3 is the pure reduction:
average, dominant, dominant_share, distinct, pixels — because a card with text
on it is not one flat colour and a bare mean of it is misleading either way.

sample_region stays strictly #if !MOB_RELEASE rather than joining screenshot's
MOB_ENABLE_SCREENSHOT opt-in: arbitrary-rect pixel reads reconstruct the screen
region by region, which would silently grant the capability that opt-in exists
to make deliberate. Pinned by a test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* clang-format ios/mob_nif.m (pre-existing violations)

Formatting only. These lines came in with the view_tree colour work earlier on
this branch (f1fb5e0, d04118c) and fail `xcrun clang-format --dry-run -Werror`,
which the pre-commit checklist runs. Kept out of the sampling commit so that
diff stays reviewable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ios): harness review fixes — alloc checks, settle caveat, post-#94 Box prose

- check enif_alloc_binary in nif_sample_region and nif_ui_paint_debug;
  return {error, alloc_failed} instead of writing through a dead binary
- document that g_ui_event_seq is process-wide (scroll notifications bump
  it too), so the 300ms settle check assumes a serial harness — an
  unrelated Mob event inside the window reads as the tap's effect
- update the Box AX prose to post-#94 reality: accessibility_role "button"
  makes Box a real AX element (.isButton) but adds no accessibilityAction,
  so accessibilityActivate still doesn't fire on_tap (verified on-sim:
  tap_xy on a role-button Box returns {error, no_effect})

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
GenericJam added a commit that referenced this pull request Sep 1, 2026
…Box prose

- check enif_alloc_binary in nif_sample_region and nif_ui_paint_debug;
  return {error, alloc_failed} instead of writing through a dead binary
- document that g_ui_event_seq is process-wide (scroll notifications bump
  it too), so the 300ms settle check assumes a serial harness — an
  unrelated Mob event inside the window reads as the tap's effect
- update the Box AX prose to post-#94 reality: accessibility_role "button"
  makes Box a real AX element (.isButton) but adds no accessibilityAction,
  so accessibilityActivate still doesn't fire on_tap (verified on-sim:
  tap_xy on a role-button Box returns {error, no_effect})

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pshoukry pushed a commit to pshoukry/mob that referenced this pull request Sep 9, 2026
* feat(ui): add intrinsic sheets and box semantics

* Address adversarial review of intrinsic sheets and box semantics

Eight findings from an adversarial pass on 15d33de, plus two of my own.

iOS Box — three interacting fixes:

- .disabled(node.disabled) replaces .allowsHitTesting(!node.disabled).
  allowsHitTesting(false) makes a view TRANSPARENT to touches, so a disabled
  box used as a blocking overlay or dimmed backdrop passed taps straight
  through to the content it was meant to be shielding. .disabled blocks
  interaction in the subtree while still consuming the touch.

- Stop branching on `disabled`. `ifLet` is @ViewBuilder if/else, i.e.
  _ConditionalContent, so flipping the branch hands SwiftUI a structurally
  different view and tears the subtree down. `disabled` is routinely toggled:
  a box wrapping a TextField lost its in-flight text and focus, and one
  wrapping a Sheet re-presented it. Tap wiring now branches on handler
  presence only, with the disabled check inside the closure, and both traits
  are unconditional OptionSet modifiers.

- Collapse to one accessibility element for a label OR an explicit button
  role, not just a label. Traits added without collapsing land on every
  descendant, so a role-only box announced each nested Text as its own button.

iOS Sheet:

- Content detents no longer flash a 1pt hairline on presentation. Content can
  only be measured once the sheet is up, so the first detent was computed from
  sentinels (intrinsic 1, environment default 1) and resolved to .height(1) on
  EVERY presentation. Measurement is now Optional and unmeasured presents at
  .medium; the environment default 0 means "root geometry unknown" and no
  longer doubles as a clamp.

- The detent accounts for the sheet's bottom safe-area inset. .height() is
  TOTAL sheet height while the content region is inset by the home indicator,
  so an intrinsic sheet was ~34pt short on notched devices — last rows under
  the indicator, and scrollable when the content was meant to fit exactly.
  The preference now carries height and inset together.

- The 0.9 ceiling is a named constant with its rationale.

Elixir:

- Restored per-rule validation messages that echo the offending value.
  [:medium, :medium] now says "must not contain duplicates, got: ..." instead
  of restating the whole grammar, and a malformed content map gets its own
  message rather than falling through to the built-in-subset branch.

- @external_resource on both tag manifests in Mob.Sigil and Mob.ScreenCase.
  They are read into module attributes at compile time, so adding Sheet
  recompiled nothing for anyone with a warm _build — every path-dep consumer,
  which is the standard mob dev setup — leaving the sigil rejecting <Sheet>
  until a manual mix clean.

- Documented the two content-detent limitations the implementation implies: a
  scrollable child expands rather than scrolling independently, and invalid
  detents now raise at the renderer boundary instead of degrading.

Test quality: the native grep tests broke on four separate behaviour-preserving
refactors here (naming a constant, unwrapping an Optional, adding the inset,
making traits unconditional). They pinned source text, not behaviour. Retargeted
at the mechanisms — the unmeasured fallback, the inset arithmetic, the
non-branching traits, .disabled over .allowsHitTesting — so they break when the
behaviour regresses rather than when the code is tidied.

1119 tests, format, credo --strict, clang-format, swiftlint (only the
pre-existing force_cast), and clang -fsyntax-only on mob_nif.m all clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Fix two Swift compile errors and verify sheets on device

Device verification found what no Elixir-side check could: PR GenericJam#94 as submitted
DID NOT BUILD for iOS. Two separate compile errors.

1. `AccessibilityTraits` has no member `.isNotEnabled`. The original
   `.accessibilityAddTraits(.isNotEnabled)` is not a real API — confirmed zero
   occurrences in the iOS 26.5 SwiftUI interface. Disabled state is not a trait
   you add; it is what `.disabled(true)` publishes to VoiceOver, which this
   code already does. Removed.

2. "The compiler is unable to type-check this expression in reasonable time."
   The accessibility and tap modifiers pushed MobBox's chain past the Swift
   type-inference budget. Extracted into a MobBoxSemantics ViewModifier — the
   same treatment MobBox itself already got from MobNodeView for the same
   reason.

Worth dwelling on how GenericJam#1 survived review: the grep test asserted
`.accessibilityAddTraits(.isNotEnabled)` appears in the source, and it did.
The string was present and the build was broken. `mix test`, `credo`,
`mix format`, `clang-format`, `swiftlint`, and `clang -fsyntax-only` on
mob_nif.m all pass on code that cannot compile, because none of them type-check
Swift. Only a native build does. The test now refutes the call form.

Device results (iPhone 17 Pro simulator, 874pt tall — chosen over the attached
iPhone SE because the SE has no home indicator and so cannot exercise the
safe-area fix at all):

- short [:content], 3 rows: hugs content, rows at y=716..784. Sheet is sized to
  content, parent visible behind.
- tall, max_height: 320, 40 rows: sheet top ~y=541, i.e. capped at ~320pt;
  content extends to y=1864 and scrolls internally.
- tall [:content], 40 rows: sheet top ~y=147, capped near the 0.9 root ceiling
  (786pt); scrolls internally.
- exact fit, 8 rows: all 8 visible, last row bottom clears the home indicator
  rather than sitting under it — the safe-area inset fix, confirmed visually.
- Sampling element_frames every 120ms from presentation showed a stable layout
  from the first sample; no degenerate frame. The 1pt case is structurally gone
  now that measurement is Optional rather than a numeric sentinel.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
pshoukry pushed a commit to pshoukry/mob that referenced this pull request Sep 9, 2026
…eadlock fix

Carries three merged changes since 0.7.31:

- GenericJam#94 intrinsic Sheet detents ([:content] / [{:content, max_height: n}]) with
  canonical validation through Mob.UI.normalize_sheet_detents!/1 re-applied at
  the renderer boundary, iOS content measurement with capping, internal
  scrolling and geometry re-clamping, composite Box accessibility (label,
  explicit button role, disabled semantics), and Sheet in both platform tag
  manifests.
- GenericJam#74 the iOS launch deadlock behind an App Store rejection — safe_area()'s
  unbounded dispatch_sync on the boot path could block the BEAM thread forever
  so the app never finished launching.
- GenericJam#78 iOS glass tint from the node background, and left-aligned fill_width rows.

The changelog records two known limitations rather than leaving them to be
discovered: a scrollable child inside a content detent expands instead of
scrolling independently (it reports intrinsic height), and a content sheet
presents at :medium for one frame before resizing, since content height is
only knowable after presentation.

Preflight: mix format, credo --strict, compile --warnings-as-errors, 1119
tests, git diff --check, and the native frame-registry harness all clean.
The Sheet work was additionally adversarially reviewed and device-verified on
an iPhone 17 Pro simulator (hug, cap, internal scroll, safe-area clearance)
and the Box/Android half on a physical Moto G.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pshoukry pushed a commit to pshoukry/mob that referenced this pull request Sep 9, 2026
…ing for colour (GenericJam#80)

* iOS harness: make tap_xy report observed effect; add painted colours to view_tree

tap_xy/2 returned ok whenever the platform input API accepted the event,
never when the app reacted. On the simulator accessibilityActivate
succeeds on a Box with on_tap while SwiftUI never runs the handler; on a
physical device the injected IOHID touch is accepted for every
coordinate and delivered for none. Both reported ok, so an agent driving
a device could not tell a working tap from a no-op.

Bump a process-wide counter from every send helper that routes a
user-originated event into the BEAM, sample it around the injection, and
poll for 300ms. ok now means the app demonstrably reacted; everything
else is a typed error (no_view_at_point / no_element_at_point /
no_effect). Hit-test up front on both paths, not just the device one.
tap_xy moves to a dirty IO scheduler since it can now block.

view_tree returned no colour at all, so a styling regression that
dropped every Box background was invisible to the one introspection API
meant to show what the device drew. Each node now carries bg_color and
text_color read back off UIView/CALayer as 0xAARRGGBB ints — the repo's
canonical colour representation (guides/theming.md). Documented the same
keys on the Android bridge contract, which no shipped MobBridge.kt
implements yet.

Fixes a latent bug the new tests surfaced: :json.decode maps JSON null to
:null, which normalize_view_tree passed straight through, so Android
labels compared unequal to nil and a null children list would have
crashed Enum.map.

Docs state the real per-platform capability instead of the aspirational
one, in Mob.Test, mob_nif.erl, CLAUDE.md and AGENTS.md.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* harness: census where paint lives; keep view_tree colour best-effort

Adds :mob_nif.ui_paint_debug/0, a census of which view/layer classes hold
readable paint, and a :class field on every view_tree node so a caller can
tell which renderer drew a node — and therefore why a colour is nil.

Extends colour extraction to the layer tree (CALayer.backgroundColor,
CAShapeLayer.fillColor, CATextLayer.foregroundColor). Measured on device
that takes it from 2 of 443 nodes to 4 of 443: SwiftUI on iOS 26 paints
through SDFLayer or rasterises into contents, so app content carries no
readable paint property at all. See the decision record — colour
verification needs screenshot sampling, not layer introspection.

* harness: sample real pixels to verify drawn colour

view_tree colour is nil for virtually all SwiftUI content on iOS 26 (4 of
443 nodes when measured), so a styling regression like the glass theme that
discarded every Box background was only findable by pixel-diffing screenshots
by hand. Sampling pixels is the only reliable answer.

New NIF sample_region/4 (iOS) returns raw RGBA for a cropped region, with the
crop done inside the render so one element's pixels cross distribution instead
of a framebuffer. It shares screenshot/3's capture path via the extracted
mob_capture_window/mob_capture_image helpers.

Mob.Test.sample_color/2 addresses a region by element :id (through
element_frames) or an explicit rect, and reduce_rgba/3 is the pure reduction:
average, dominant, dominant_share, distinct, pixels — because a card with text
on it is not one flat colour and a bare mean of it is misleading either way.

sample_region stays strictly #if !MOB_RELEASE rather than joining screenshot's
MOB_ENABLE_SCREENSHOT opt-in: arbitrary-rect pixel reads reconstruct the screen
region by region, which would silently grant the capability that opt-in exists
to make deliberate. Pinned by a test.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* clang-format ios/mob_nif.m (pre-existing violations)

Formatting only. These lines came in with the view_tree colour work earlier on
this branch (f1fb5e0, d04118c) and fail `xcrun clang-format --dry-run -Werror`,
which the pre-commit checklist runs. Kept out of the sampling commit so that
diff stays reviewable.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ios): harness review fixes — alloc checks, settle caveat, post-GenericJam#94 Box prose

- check enif_alloc_binary in nif_sample_region and nif_ui_paint_debug;
  return {error, alloc_failed} instead of writing through a dead binary
- document that g_ui_event_seq is process-wide (scroll notifications bump
  it too), so the 300ms settle check assumes a serial harness — an
  unrelated Mob event inside the window reads as the tap's effect
- update the Box AX prose to post-GenericJam#94 reality: accessibility_role "button"
  makes Box a real AX element (.isButton) but adds no accessibilityAction,
  so accessibilityActivate still doesn't fire on_tap (verified on-sim:
  tap_xy on a role-button Box returns {error, no_effect})

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant