Skip to content

Add iOS native intelligence bridges - #8

Closed
YurkoHoshko wants to merge 6 commits into
GenericJam:masterfrom
YurkoHoshko:codex-native-ai-ios
Closed

Add iOS native intelligence bridges#8
YurkoHoshko wants to merge 6 commits into
GenericJam:masterfrom
YurkoHoshko:codex-native-ai-ios

Conversation

@YurkoHoshko

@YurkoHoshko YurkoHoshko commented May 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Summary

  • adds iOS-native intelligence bridges for Foundation Models text generation, Vision text recognition, and Speech file transcription
  • exposes the Elixir API by Apple SDK boundary under Mob.IOS.FoundationModels, Mob.IOS.Vision, and Mob.IOS.Speech
  • registers matching NIF names on iOS and Android, with Android returning :unsupported
  • documents usage, Apple SDK references, current scope, known limitations, and what maintainers can test in simulator vs on device
  • documents smoke tests using Mob.Photos.pick/2 for OCR input and Mob.Audio.start_recording/2 / Mob.Audio.stop_recording/1 for Speech input

AI-assisted disclosure

This PR was implemented with AI assistance from Codex. The changes were reviewed, simplified, and validated locally before pushing.

Where to test

Capability iOS simulator Physical iPhone
Mob.IOS.FoundationModels.generate_text/3 Not expected to work. The simulator does not provide the on-device system language model. Expected to work on Apple Intelligence-capable devices with Apple Intelligence enabled and the model ready.
Mob.IOS.Vision.recognize_text/3 Expected to work. Use Mob.Photos.pick/2 to choose a simulator photo, then pass the selected path to Mob.IOS.Vision.recognize_text/3. I also verified OCR against a PNG copied into the simulator app container. Expected to work.
Mob.IOS.Speech.transcribe_audio/3 Expected to reach the Speech bridge after recording with Mob.Audio.start_recording/2 and Mob.Audio.stop_recording/1. The simulator may still return recognizer initialization errors depending on Speech authorization, locale, and runtime service availability. Expected to work, subject to Speech authorization and locale support.

Validation

  • mix format
  • mix test
  • mix compile --warnings-as-errors
  • mix docs
  • xcrun --sdk iphonesimulator swiftc -parse -target arm64-apple-ios26.0-simulator ios/MobFoundationModels.swift ios/MobVision.swift ios/MobSpeech.swift
  • simulator deployment of ios_ai_lab against branch commit 37fd807
  • fresh simulator app deployment
ScreenRecording_05-17-2026.00-51-38_1.MP4

@YurkoHoshko YurkoHoshko changed the title Add iOS native AI bridge [WIP] Add iOS native AI bridge May 15, 2026
@YurkoHoshko YurkoHoshko changed the title [WIP] Add iOS native AI bridge Add iOS native intelligence bridges May 15, 2026
YurkoHoshko
YurkoHoshko commented May 15, 2026
View reviewed changes
Comment thread lib/mob/foundation_models.ex Outdated
YurkoHoshko
YurkoHoshko commented May 16, 2026
View reviewed changes
Comment thread guides/native_intelligence.md Outdated
@YurkoHoshko
YurkoHoshko marked this pull request as ready for review May 17, 2026 07:53
@dl-alexandre dl-alexandre mentioned this pull request May 19, 2026
Open
@YurkoHoshko

YurkoHoshko commented May 22, 2026

Copy link
Copy Markdown
Author

closing to be re-built as plugin

pshoukry pushed a commit to pshoukry/mob that referenced this pull request Sep 9, 2026
@GenericJam
ios: AX modifiers on Slider + Toggle so Mob.Test can drive them
6ceea5c 
Closes mob/issues.md GenericJam#7 + GenericJam#8 — both were AX-conformance bugs in the
SwiftUI components that backed `<Slider>` and `<Toggle>` in the
renderer.

GenericJam#7: `MobSlider` had no `.accessibilityAdjustableAction`. SwiftUI
ignores VoiceOver / `accessibilityIncrement` calls without it; the
NIF returned :ok but the value never moved. Added the modifier with
default step `(max - min) / 10` (matches VoiceOver's default for
native UISlider) and a wired callback so on-change events fire too.

GenericJam#8: `MobToggle` used SwiftUI's `Toggle("Label", isOn:)` form, which
does NOT propagate the label string into the underlying control's
`accessibilityLabel`. The AX tree exposed the toggle as a button
with empty label, so `Mob.Test.toggle(node, "Notifications")`
returned :label_not_found. Added `.accessibilityLabel(label)` after
the toggle so the visible text reaches the AX tree.

Both fixes are in `ios/MobRootView.swift`. Android counterparts
(`MobSlider` / `MobToggle` in MobBridgeKt) need analogous
`Modifier.semantics { setProgress(...) / contentDescription = ... }`
calls — tracked separately under issues.md GenericJam#11 (Compose semantics).
pshoukry pushed a commit to pshoukry/mob that referenced this pull request Sep 9, 2026
@GenericJam
issues: mark GenericJam#7, GenericJam#8 fixed and GenericJam#14 worke…
34b7e3d 
…d-around

GenericJam#7 + GenericJam#8 — `MobSlider` and `MobToggle` in MobRootView.swift gained the
SwiftUI AX modifiers SwiftUI doesn't apply by default
(.accessibilityAdjustableAction with default step (max-min)/10 for
the slider; .accessibilityLabel(label) for the toggle). Mob.Test
end-to-end driving works after these.

GenericJam#14 — defensive fallback in MobDev.Connector. When the primary node
name times out the connector now also tries the alternate
`<app>_ios@127.0.0.1` form (without the udid suffix). The connected
Device.node is updated to whichever responded so downstream RPC
calls use the correct address. The root cause (mob_beam.m sometimes
not seeing SIMULATOR_UDID) is captured in the resolution note for a
future investigation.

Remaining: GenericJam#3 (WebSocket→longpoll, investigation), GenericJam#6 (16 KB page
alignment, OTP rebuild), GenericJam#9 (Alert OK button, UIKit deep dive),
GenericJam#10 (Android 17 SELinux, OTP source patch), GenericJam#11 (Android Compose
semantics walker, ~200 lines Kotlin).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@YurkoHoshko